Thread: First draft of PG 17 release notes
I have committed the first draft of the PG 17 release notes; you can
see the results here:
https://momjian.us/pgsql_docs/release-17.html
It will be improved until the final release. The item count is 188,
which is similar to recent releases:
release-10: 189
release-11: 170
release-12: 180
release-13: 178
release-14: 220
release-15: 184
release-16: 206
release-17: 188
I welcome feedback. For some reason it was an easier job than usual.
--
Bruce Momjian <bruce@momjian.us> https://momjian.us
EDB https://enterprisedb.com
Only you can decide what is important to you.
On Thu, 9 May 2024 at 16:04, Bruce Momjian <bruce@momjian.us> wrote: > I welcome feedback. For some reason it was an easier job than usual. Thanks for working on that. > +2023-11-02 [cac169d68] Increase DEFAULT_FDW_TUPLE_COST from 0.01 to 0.2 > +Double the default foreign data wrapper tuple cost (David Rowley, Umair Shahid) That's 20x rather than 2x. David
Hi Bruce,
A minor formatting issue in the start below. Bullet is not required here.
On Thu, May 9, 2024 at 9:45 AM David Rowley <dgrowleyml@gmail.com> wrote:
On Thu, 9 May 2024 at 16:04, Bruce Momjian <bruce@momjian.us> wrote:
> I welcome feedback. For some reason it was an easier job than usual.
Thanks for working on that.
> +2023-11-02 [cac169d68] Increase DEFAULT_FDW_TUPLE_COST from 0.01 to 0.2
> +Double the default foreign data wrapper tuple cost (David Rowley, Umair Shahid)
That's 20x rather than 2x.
David
Muhammad Ikram
On Thu, 9 May 2024 at 16:47, Muhammad Ikram <mmikram@gmail.com> wrote: > A minor formatting issue in the start below. Bullet is not required here. This is a placeholder for the highlight features of v17 will go. Bruce tends not to decide what those are all by himself. David
Hi, On Thu, May 09, 2024 at 12:03:50AM -0400, Bruce Momjian wrote: > I have committed the first draft of the PG 17 release notes; you can > see the results here: > > https://momjian.us/pgsql_docs/release-17.html Thanks for working on that! > I welcome feedback. > Add system view pg_wait_events that reports wait event types (Michael Paquier) Michael is the committer for 1e68e43d3f, the author is me. Regards, -- Bertrand Drouvot PostgreSQL Contributors Team RDS Open Source Databases Amazon Web Services: https://aws.amazon.com
Hi, On Thu, May 9, 2024 at 1:03 PM Bruce Momjian <bruce@momjian.us> wrote: > > I have committed the first draft of the PG 17 release notes; you can > see the results here: > > https://momjian.us/pgsql_docs/release-17.html Thank you for working on that! I'd like to mention some of my works. I think we can add the vacuum performance improvements by the following commits: - Add template for adaptive radix tree (ee1b30f1) - Add TIDStore, to store sets of TIDs (ItemPointerData) efficiently (30e144287) - Use TidStore for dead tuple TIDs storage during lazy vacuum (667e65aac) Also, please consider the following item: - Improve eviction algorithm in ReorderBuffer using max-heap for many subtransactions (5bec1d6bc) Finally, should we mention the following commit in the release note? It's not a user-visible change but added a new regression test module. - Add tests for XID wraparound (e255b646a) Regards, -- Masahiko Sawada Amazon Web Services: https://aws.amazon.com
On Thu, May 9, 2024 at 12:04 PM Bruce Momjian <bruce@momjian.us> wrote:
I have committed the first draft of the PG 17 release notes; you can
see the results here:
https://momjian.us/pgsql_docs/release-17.html
Thanks for working on that.
For this item:
Allow the optimizer to improve CTE plans by using the sort order of
columns referenced in earlier CTE clauses (Jian Guo)
I think you mean a65724dfa. The author should be 'Richard Guo'.
And I'm wondering if it is more accurate to state it as "Allow the
optimizer to improve plans for the outer query by leveraging the sort
order of a CTE's output."
I think maybe a similar revision can be applied to the item just above
this one.
Thanks
Richard
And I'm wondering if it is more accurate to state it as "Allow the
optimizer to improve plans for the outer query by leveraging the sort
order of a CTE's output."
I think maybe a similar revision can be applied to the item just above
this one.
Thanks
Richard
Hi, > I have committed the first draft of the PG 17 release notes; you can > see the results here: > > https://momjian.us/pgsql_docs/release-17.html > > It will be improved until the final release. The item count is 188, > which is similar to recent releases: Thanks for working on this. I believe the part of the 64-bit XIDs patchset that was delivered in PG17 is worth highlighting in "E.1.3.10. Source Code" section: 4ed8f0913bfd 2cdf131c46e6 5a1dfde8334b a60b8a58f435 All this can probably be summarized as one bullet "Index SLRUs by 64-bit integers rather than by 32-bit ones" where the authors are: Maxim Orlov, Aleksander Alekseev, Alexander Korotkov, Teodor Sigaev, Nikita Glukhov, Pavel Borisov, Yura Sokolov. -- Best regards, Aleksander Alekseev
On Thu, May 9, 2024 at 12:04 PM Bruce Momjian <bruce@momjian.us> wrote: > > I have committed the first draft of the PG 17 release notes; you can > see the results here: > > https://momjian.us/pgsql_docs/release-17.html > another potential incompatibilities issue: ALTER TABLE DROP PRIMARY KEY see: https://www.postgresql.org/message-id/202404181849.6frtmajobe27%40alvherre.pgsql
Bruce Momjian <bruce@momjian.us> writes: > I have committed the first draft of the PG 17 release notes; you can > see the results here: > > https://momjian.us/pgsql_docs/release-17.html My name is listed twice in the "Improve psql tab completion" item. - ilmari
Dagfinn Ilmari Mannsåker <ilmari@ilmari.org> writes: > Bruce Momjian <bruce@momjian.us> writes: > >> I have committed the first draft of the PG 17 release notes; you can >> see the results here: >> >> https://momjian.us/pgsql_docs/release-17.html > > My name is listed twice in the "Improve psql tab completion" item. You can move one of them to "Track DEALLOCATE in pg_stat_statements", which Michael and I co-authored. - ilmari
On Thu, May 9, 2024 at 12:04 PM Bruce Momjian <bruce@momjian.us> wrote: > I have committed the first draft of the PG 17 release notes; you can > see the results here: > > https://momjian.us/pgsql_docs/release-17.html > * Add function pg_buffercache_evict() to allow shared buffer eviction (Palak Chaturvedi, Thomas Munro) * This is useful for testing. this should put it on the section < E.1.3.11. Additional Modules ? Then I found out official release notes don't have <section> attributes, so it doesn't matter? << Allow ALTER OPERATOR to set more optimization attributes (Tommy Pavlicek) This is useful for extensions. << I think this commit title "Add hash support functions and hash opclass for contrib/ltree." from [1] is more descriptive. i am not 100% sure of the meaning of "This is useful for extensions." [1] https://git.postgresql.org/cgit/postgresql.git/commit/?id=485f0aa85995340fb62113448c992ee48dc6fff1
> << > Allow ALTER OPERATOR to set more optimization attributes (Tommy Pavlicek) > This is useful for extensions. > << sorry, I mean << Allow the creation of hash indexes on ltree columns (Tommy Pavlicek) This also enables hash join and hash aggregation on ltree columns. << better description would be: << Add hash support functions and hash opclass for contrib/ltree (Tommy Pavlicek) This also enables hash join and hash aggregation on ltree columns. <<
Some findings
1.
>>Remove adminpack contrib extension (Daniel Gustafsson)
>>This was used by non-end-of-life pgAdmin III.
Perhaps you mean now-end-of-life (s/non/now/)
2.
>>All specification of partitioned table access methods (Justin Pryzby, >>Soumyadeep Chakraborty, Michael Paquier)
perhaps you mean Allow, otherwise meaning not clear.
3.
>> Add some long options to pg_archivecleanup (Atsushi Torikoshi)
>>The long options are --debug, --dry-run, and /--strip-extension.
The slash should be omitted.
Hans Buschmann
On Thu, May 9, 2024 at 6:53 PM jian he <jian.universality@gmail.com> wrote: > > On Thu, May 9, 2024 at 12:04 PM Bruce Momjian <bruce@momjian.us> wrote: > > I have committed the first draft of the PG 17 release notes; you can > > see the results here: > > > > https://momjian.us/pgsql_docs/release-17.html < Add columns to pg_stats to report range histogram information (Egor Rogov, Soumyadeep Chakraborty) I think this applies to range type and multi range type, "range histogram information" seems not very clear to me. So maybe: < Add columns to pg_stats to report range-type histogram information (Egor Rogov, Soumyadeep Chakraborty) Display length and bounds histograms in pg_stats < Add new COPY option "ON_ERROR ignore" to discard error rows (Damir Belyalov, Atsushi Torikoshi, Alex Shulgin, Jian He, Jian He, Yugo Nagata) duplicate name.
On Thu, May 9, 2024 at 04:44:47PM +1200, David Rowley wrote:
> On Thu, 9 May 2024 at 16:04, Bruce Momjian <bruce@momjian.us> wrote:
> > I welcome feedback. For some reason it was an easier job than usual.
>
> Thanks for working on that.
>
> > +2023-11-02 [cac169d68] Increase DEFAULT_FDW_TUPLE_COST from 0.01 to 0.2
>
> > +Double the default foreign data wrapper tuple cost (David Rowley, Umair Shahid)
>
> That's 20x rather than 2x.
Oops, changed to:
Increase the default foreign data wrapper tuple cost (David
Rowley, Umair Shahid)
--
Bruce Momjian <bruce@momjian.us> https://momjian.us
EDB https://enterprisedb.com
Only you can decide what is important to you.
On Thu, May 9, 2024 at 09:47:34AM +0500, Muhammad Ikram wrote:
> Hi Bruce,
>
> A minor formatting issue in the start below. Bullet is not required here.
>
>
> E.1.1. Overview
>
> PostgreSQL 17 contains many new features and enhancements, including:
>
> •
>
> The above items and other new features of PostgreSQL 17 are explained in more
> detail in the sections below.
That is just a place-holder. I changed the bullet text to be:
TO BE COMPLETED LATER
--
Bruce Momjian <bruce@momjian.us> https://momjian.us
EDB https://enterprisedb.com
Only you can decide what is important to you.
On Thu, May 9, 2024 at 04:52:14PM +1200, David Rowley wrote: > On Thu, 9 May 2024 at 16:47, Muhammad Ikram <mmikram@gmail.com> wrote: > > A minor formatting issue in the start below. Bullet is not required here. > > This is a placeholder for the highlight features of v17 will go. > Bruce tends not to decide what those are all by himself. Yes, I already have so much of my opinion in the release notes that I prefer others to make that list, and to make the Acknowledgments list at the bottom. -- Bruce Momjian <bruce@momjian.us> https://momjian.us EDB https://enterprisedb.com Only you can decide what is important to you.
On Thu, May 9, 2024 at 04:53:38AM +0000, Bertrand Drouvot wrote: > Hi, > > On Thu, May 09, 2024 at 12:03:50AM -0400, Bruce Momjian wrote: > > I have committed the first draft of the PG 17 release notes; you can > > see the results here: > > > > https://momjian.us/pgsql_docs/release-17.html > > Thanks for working on that! > > > I welcome feedback. > > > Add system view pg_wait_events that reports wait event types (Michael Paquier) > > Michael is the committer for 1e68e43d3f, the author is me. Wow, thank you for finding that. The commit message is very clear so I don't know how I made that mistake. Fixed. -- Bruce Momjian <bruce@momjian.us> https://momjian.us EDB https://enterprisedb.com Only you can decide what is important to you.
On Thu, May 9, 2024 at 02:17:12PM +0900, Masahiko Sawada wrote: > Hi, > > On Thu, May 9, 2024 at 1:03 PM Bruce Momjian <bruce@momjian.us> wrote: > > > > I have committed the first draft of the PG 17 release notes; you can > > see the results here: > > > > https://momjian.us/pgsql_docs/release-17.html > > Thank you for working on that! > > I'd like to mention some of my works. I think we can add the vacuum > performance improvements by the following commits: > > - Add template for adaptive radix tree (ee1b30f1) > - Add TIDStore, to store sets of TIDs (ItemPointerData) efficiently (30e144287) > - Use TidStore for dead tuple TIDs storage during lazy vacuum (667e65aac) Okay, I reworded the item, added authors, and added the commits: <!-- Author: John Naylor <john.naylor@postgresql.org> 2024-03-07 [ee1b30f12] Add template for adaptive radix tree Author: Masahiko Sawada <msawada@postgresql.org> 2024-03-21 [30e144287] Add TIDStore, to store sets of TIDs (ItemPointerData) ef Author: Masahiko Sawada <msawada@postgresql.org> 2024-04-02 [667e65aac] Use TidStore for dead tuple TIDs storage during lazy vac Author: Heikki Linnakangas <heikki.linnakangas@iki.fi> 2024-04-03 [6dbb49026] Combine freezing and pruning steps in VACUUM --> <listitem> <para> Allow vacuum to more efficiently remove and freeze tuples (John Naylor, Masahiko Sawada, Melanie Plageman) </para> </listitem> > Also, please consider the following item: > > - Improve eviction algorithm in ReorderBuffer using max-heap for many > subtransactions (5bec1d6bc) I looked at that item and I don't have a generic "make logical replication apply faster" item to merge it into, and many subtransactions seemed like enough of an edge-case that I didn't think mentioning it make sense. Can you see a good place to add it? > Finally, should we mention the following commit in the release note? > It's not a user-visible change but added a new regression test module. > > - Add tests for XID wraparound (e255b646a) I don't normally add testing infrastructure changes unless they are major. -- Bruce Momjian <bruce@momjian.us> https://momjian.us EDB https://enterprisedb.com Only you can decide what is important to you.
On Thu, May 9, 2024 at 02:37:57PM +0800, Richard Guo wrote: > > On Thu, May 9, 2024 at 12:04 PM Bruce Momjian <bruce@momjian.us> wrote: > > I have committed the first draft of the PG 17 release notes; you can > see the results here: > > https://momjian.us/pgsql_docs/release-17.html > > > Thanks for working on that. > > For this item: > > > Allow the optimizer to improve CTE plans by using the sort order of > columns referenced in earlier CTE clauses (Jian Guo) > > > I think you mean a65724dfa. The author should be 'Richard Guo'. Wow the CTE item above it was done by Jian Guo. I probably copied the text from the line above it, modified the description, but thought the author's name was the same, but it was not. Fixed. > And I'm wondering if it is more accurate to state it as "Allow the > optimizer to improve plans for the outer query by leveraging the sort > order of a CTE's output." > > I think maybe a similar revision can be applied to the item just above > this one. Okay, I went with this text: <!-- Author: Tom Lane <tgl@sss.pgh.pa.us> 2023-11-17 [f7816aec2] Extract column statistics from CTE references, if possib --> <listitem> <para> Allow the optimizer to improve CTE plans by considering the statistics of columns referenced in earlier row output clauses(Jian Guo, Tom Lane) </para> </listitem> <!-- Author: Tom Lane <tgl@sss.pgh.pa.us> 2024-03-26 [a65724dfa] Propagate pathkeys from CTEs up to the outer query. --> <listitem> <para> Allow the optimizer to improve CTE plans by considering the sort order of columns referenced in earlier row output clauses(Richard Guo) </para> </listitem> I did not use "leveraging" because I am concerned non-native English speakers might find the term confusing. -- Bruce Momjian <bruce@momjian.us> https://momjian.us EDB https://enterprisedb.com Only you can decide what is important to you.
On Thu, May 9, 2024 at 12:18:44PM +0300, Aleksander Alekseev wrote: > Hi, > > > I have committed the first draft of the PG 17 release notes; you can > > see the results here: > > > > https://momjian.us/pgsql_docs/release-17.html > > > > It will be improved until the final release. The item count is 188, > > which is similar to recent releases: > > Thanks for working on this. > > I believe the part of the 64-bit XIDs patchset that was delivered in > PG17 is worth highlighting in "E.1.3.10. Source Code" section: > > 4ed8f0913bfd > 2cdf131c46e6 > 5a1dfde8334b > a60b8a58f435 > > All this can probably be summarized as one bullet "Index SLRUs by > 64-bit integers rather than by 32-bit ones" where the authors are: > Maxim Orlov, Aleksander Alekseev, Alexander Korotkov, Teodor Sigaev, > Nikita Glukhov, Pavel Borisov, Yura Sokolov. Wow, I try to only list source code items that have some user-facing impact, and I don't think these do. I do realize how important they are though. This gets into the balance of mentioning items _users_ need to know about, vs. important improvements that _we_ know about. -- Bruce Momjian <bruce@momjian.us> https://momjian.us EDB https://enterprisedb.com Only you can decide what is important to you.
On Thu, May 9, 2024 at 06:00:24PM +0800, jian he wrote: > On Thu, May 9, 2024 at 12:04 PM Bruce Momjian <bruce@momjian.us> wrote: > > > > I have committed the first draft of the PG 17 release notes; you can > > see the results here: > > > > https://momjian.us/pgsql_docs/release-17.html > > > > another potential incompatibilities issue: > ALTER TABLE DROP PRIMARY KEY > > see: > https://www.postgresql.org/message-id/202404181849.6frtmajobe27%40alvherre.pgsql I see it now, and I see Alvaro Herrera saying: https://www.postgresql.org/message-id/202404181849.6frtmajobe27%40alvherre.pgsql > I wonder is there any incompatibility issue, or do we need to say something > about the new behavior when dropping a key column? --> Umm, yeah, maybe we should document it in ALTER TABLE DROP PRIMARY KEY --> and in the release notes to note the different behavior. However, I don't see it mentioned as a release note item in the commit message or mentioned in our docs. I suppose the release note text would be: Removing a PRIMARY KEY will remove the NOT NULL column specification Previously the NOT NULL specification would be retained. Do we have agreement that we want this release note item? -- Bruce Momjian <bruce@momjian.us> https://momjian.us EDB https://enterprisedb.com Only you can decide what is important to you.
On Thu, May 9, 2024 at 11:22:06AM +0100, Dagfinn Ilmari Mannsåker wrote: > Bruce Momjian <bruce@momjian.us> writes: > > > I have committed the first draft of the PG 17 release notes; you can > > see the results here: > > > > https://momjian.us/pgsql_docs/release-17.html > > My name is listed twice in the "Improve psql tab completion" item. You did such a great job I wanted to list you twice. :-) Actually, the author list was so long I just didn't notice, fixed. -- Bruce Momjian <bruce@momjian.us> https://momjian.us EDB https://enterprisedb.com Only you can decide what is important to you.
On Thu, May 9, 2024 at 11:31:17AM +0100, Dagfinn Ilmari Mannsåker wrote: > Dagfinn Ilmari Mannsåker <ilmari@ilmari.org> writes: > > > Bruce Momjian <bruce@momjian.us> writes: > > > >> I have committed the first draft of the PG 17 release notes; you can > >> see the results here: > >> > >> https://momjian.us/pgsql_docs/release-17.html > > > > My name is listed twice in the "Improve psql tab completion" item. > > You can move one of them to "Track DEALLOCATE in pg_stat_statements", > which Michael and I co-authored. Yep, also my mistake, fixed. My apologies. -- Bruce Momjian <bruce@momjian.us> https://momjian.us EDB https://enterprisedb.com Only you can decide what is important to you.
On Thu, May 9, 2024 at 06:53:30PM +0800, jian he wrote: > On Thu, May 9, 2024 at 12:04 PM Bruce Momjian <bruce@momjian.us> wrote: > > I have committed the first draft of the PG 17 release notes; you can > > see the results here: > > > > https://momjian.us/pgsql_docs/release-17.html > > > > * Add function pg_buffercache_evict() to allow shared buffer eviction > (Palak Chaturvedi, Thomas Munro) > * This is useful for testing. > > this should put it on the section > < E.1.3.11. Additional Modules > ? Oh, it is in the pg_buffercache module --- I should have realized that from the name, fixed. > Then I found out official release notes don't have <section> attributes, > so it doesn't matter? Uh, what are sections? Did previous release notes have it? > I think this commit title "Add hash support functions and hash opclass > for contrib/ltree." > from [1] is more descriptive. Uh, I don't think people know what hash support functions are, but they know what hash indexes are, and maybe hash joins and hash aggregates. Why do you consider the commit text better? > i am not 100% sure of the meaning of "This is useful for extensions." The commit says: commit 2b5154beab7 Author: Tom Lane <tgl@sss.pgh.pa.us> Date: Fri Oct 20 12:28:38 2023 -0400 Extend ALTER OPERATOR to allow setting more optimization attributes. Allow the COMMUTATOR, NEGATOR, MERGES, and HASHES attributes to be set by ALTER OPERATOR. However, we don't allow COMMUTATOR/NEGATOR to be changed once set, nor allow the MERGES/HASHES flags to be unset once set. Changes like that might invalidate plans already made, and dealing with the consequences seems like more trouble than it's worth. --> The main use-case we foresee for this is to allow addition of missed --> properties in extension update scripts, such as extending an existing --> operator to support hashing. So only transitions from not-set to set states seem very useful. -- Bruce Momjian <bruce@momjian.us> https://momjian.us EDB https://enterprisedb.com Only you can decide what is important to you.
On Thu, May 9, 2024 at 06:57:01PM +0800, jian he wrote: > > << > > Allow ALTER OPERATOR to set more optimization attributes (Tommy Pavlicek) > > This is useful for extensions. > > << > > sorry, I mean > << > Allow the creation of hash indexes on ltree columns (Tommy Pavlicek) > This also enables hash join and hash aggregation on ltree columns. > << > > better description would be: > << > Add hash support functions and hash opclass for contrib/ltree (Tommy Pavlicek) > This also enables hash join and hash aggregation on ltree columns. > << Yes, please see my previous email where I am asking why being more specific is worse. -- Bruce Momjian <bruce@momjian.us> https://momjian.us EDB https://enterprisedb.com Only you can decide what is important to you.
On Thu, May 9, 2024 at 07:49:55PM +0800, jian he wrote: > On Thu, May 9, 2024 at 6:53 PM jian he <jian.universality@gmail.com> wrote: > > > > On Thu, May 9, 2024 at 12:04 PM Bruce Momjian <bruce@momjian.us> wrote: > > > I have committed the first draft of the PG 17 release notes; you can > > > see the results here: > > > > > > https://momjian.us/pgsql_docs/release-17.html > > < Add columns to pg_stats to report range histogram information (Egor > Rogov, Soumyadeep Chakraborty) > I think this applies to range type and multi range type, "range > histogram information" seems not very clear to me. > So maybe: > < Add columns to pg_stats to report range-type histogram information > (Egor Rogov, Soumyadeep Chakraborty) Yes, good point, done. > Display length and bounds histograms in pg_stats Uh, isn't that assumed? Is this a detail worth mentioning? > < Add new COPY option "ON_ERROR ignore" to discard error rows (Damir > Belyalov, Atsushi Torikoshi, Alex Shulgin, Jian He, Jian He, Yugo > Nagata) > duplicate name. Fixed. -- Bruce Momjian <bruce@momjian.us> https://momjian.us EDB https://enterprisedb.com Only you can decide what is important to you.
On Thu, May 9, 2024 at 11:12 PM Bruce Momjian <bruce@momjian.us> wrote: > > On Thu, May 9, 2024 at 07:49:55PM +0800, jian he wrote: > > On Thu, May 9, 2024 at 6:53 PM jian he <jian.universality@gmail.com> wrote: > > > > > > On Thu, May 9, 2024 at 12:04 PM Bruce Momjian <bruce@momjian.us> wrote: > > > > I have committed the first draft of the PG 17 release notes; you can > > > > see the results here: > > > > > > > > https://momjian.us/pgsql_docs/release-17.html > > E.1.3.1.5. Privileges Add per-table GRANT permission MAINTAIN to control maintenance operations (Nathan Bossart) The operations are VACUUM, ANALYZE, REINDEX, REFRESH MATERIALIZE VIEW, CLUSTER, and LOCK TABLE. Add user-grantable role pg_maintain to control maintenance operations (Nathan Bossart) The operations are VACUUM, ANALYZE, REINDEX, REFRESH MATERIALIZE VIEW, CLUSTER, and LOCK TABLE. Allow roles with pg_monitor privileges to execute pg_current_logfile() (Pavlo Golub, Nathan Bossart) --------------- should be "REFRESH MATERIALIZED VIEW"? also "Allow roles with pg_monitor privileges to execute pg_current_logfile() (Pavlo Golub, Nathan Bossart)" "pg_monitor" is a predefined role, so technically, "with pg_monitor privileges" is not correct? -------------------------------------------------------------------------- Add function XMLText() to convert text to a single XML text node (Jim Jones) XMLText() should be xmltext() -------------------------------------------------------------------------- Add function to_regtypemod() to return the typemod of a string (David Wheeler, Erik Wienhold) I think this description does not mean the same thing as the doc[1] [1] https://www.postgresql.org/docs/devel/functions-info.html#FUNCTIONS-INFO-CATALOG -------------------------------------------------------------------------- Allow GROUP BY columns to be internally ordered to match ORDER BY (Andrei Lepikhov, Teodor Sigaev) This can be disabled using server variable enable_group_by_reordering. Probably `This can be disabled by setting the server variable enable_group_by_reordering to false`.
On Thu, May 9, 2024 at 11:03:39AM +0000, Hans Buschmann wrote:
> Some findings
>
>
> 1.
>
> >>Remove adminpack contrib extension (Daniel Gustafsson)
>
> >>This was used by non-end-of-life pgAdmin III.
>
>
> Perhaps you mean now-end-of-life (s/non/now/)
Yes, fixed to "now end-of-life"
> 2.
> >>All specification of partitioned table access methods (Justin Pryzby, >>
> Soumyadeep Chakraborty, Michael Paquier)
>
> perhaps you mean Allow, otherwise meaning not clear.
Yep, fixed.
> 3.
> >> Add some long options to pg_archivecleanup (Atsushi Torikoshi)
> >>The long options are --debug, --dry-run, and /--strip-extension.
>
> The slash should be omitted.
Fixed, not sure how that got in there.
I have committed all outstanding fixes and updated the doc build:
https://momjian.us/pgsql_docs/release-17.html
Thank you for all the valuable feedback.
Incidentally, the big surprise for me was the large number of optimizer
improvements.
--
Bruce Momjian <bruce@momjian.us> https://momjian.us
EDB https://enterprisedb.com
Only you can decide what is important to you.
On Thu, May 9, 2024 at 11:26:44PM +0800, jian he wrote: > On Thu, May 9, 2024 at 11:12 PM Bruce Momjian <bruce@momjian.us> wrote: > > > > On Thu, May 9, 2024 at 07:49:55PM +0800, jian he wrote: > > > On Thu, May 9, 2024 at 6:53 PM jian he <jian.universality@gmail.com> wrote: > > > > > > > > On Thu, May 9, 2024 at 12:04 PM Bruce Momjian <bruce@momjian.us> wrote: > > > > > I have committed the first draft of the PG 17 release notes; you can > > > > > see the results here: > > > > > > > > > > https://momjian.us/pgsql_docs/release-17.html > > > > > E.1.3.1.5. Privileges > Add per-table GRANT permission MAINTAIN to control maintenance > operations (Nathan Bossart) > > The operations are VACUUM, ANALYZE, REINDEX, REFRESH MATERIALIZE VIEW, > CLUSTER, and LOCK TABLE. > > Add user-grantable role pg_maintain to control maintenance operations > (Nathan Bossart) > > The operations are VACUUM, ANALYZE, REINDEX, REFRESH MATERIALIZE VIEW, > CLUSTER, and LOCK TABLE. > > Allow roles with pg_monitor privileges to execute pg_current_logfile() > (Pavlo Golub, Nathan Bossart) > --------------- > should be "REFRESH MATERIALIZED VIEW"? Yes, fixed. > also > "Allow roles with pg_monitor privileges to execute > pg_current_logfile() (Pavlo Golub, Nathan Bossart)" > "pg_monitor" is a predefined role, so technically, "with pg_monitor > privileges" is not correct? Good point, new text: Allow roles with pg_monitor membership to execute pg_current_logfile() (Pavlo Golub, Nathan Bossart) > -------------------------------------------------------------------------- > Add function XMLText() to convert text to a single XML text node (Jim Jones) > > XMLText() > should be > xmltext() Right, fixed. > -------------------------------------------------------------------------- > Add function to_regtypemod() to return the typemod of a string (David > Wheeler, Erik Wienhold) > I think this description does not mean the same thing as the doc[1] Yes, I see your point. I changed the text to: Add function to_regtypemod() to return the type modifier of a type specification (David Wheeler, Erik Wienhold) > [1] https://www.postgresql.org/docs/devel/functions-info.html#FUNCTIONS-INFO-CATALOG > -------------------------------------------------------------------------- > > Allow GROUP BY columns to be internally ordered to match ORDER BY > (Andrei Lepikhov, Teodor Sigaev) > This can be disabled using server variable enable_group_by_reordering. > > Probably > `This can be disabled by setting the server variable > enable_group_by_reordering to false`. Uh, I usually don't go into that detail. There will be a link to the variable in about a month so users can look up its behavior. -- Bruce Momjian <bruce@momjian.us> https://momjian.us EDB https://enterprisedb.com Only you can decide what is important to you.
On 2024-05-09 Th 00:03, Bruce Momjian wrote:
I have committed the first draft of the PG 17 release notes; you can see the results here: https://momjian.us/pgsql_docs/release-17.html It will be improved until the final release. The item count is 188, which is similar to recent releases: release-10: 189 release-11: 170 release-12: 180 release-13: 178 release-14: 220 release-15: 184 release-16: 206 release-17: 188 I welcome feedback. For some reason it was an easier job than usual.
Remove the ability to build Postgres with Visual Studio (Michael Paquier)
Meson is now the only available Windows build method.
This is a category mistake. What was removed was the special code we had for building with VS, but not the ability to build with VS. You can build with VS using meson (see for example drongo on the buildfarm)
cheers
andrew
-- Andrew Dunstan EDB: https://www.enterprisedb.com
On Thu, May 9, 2024 at 12:10:11PM -0400, Andrew Dunstan wrote: > > On 2024-05-09 Th 00:03, Bruce Momjian wrote: > > I have committed the first draft of the PG 17 release notes; you can > see the results here: > > https://momjian.us/pgsql_docs/release-17.html > > It will be improved until the final release. The item count is 188, > which is similar to recent releases: > > release-10: 189 > release-11: 170 > release-12: 180 > release-13: 178 > release-14: 220 > release-15: 184 > release-16: 206 > release-17: 188 > > I welcome feedback. For some reason it was an easier job than usual. > > > • Remove the ability to build Postgres with Visual Studio (Michael Paquier) > > Meson is now the only available Windows build method. > > > This is a category mistake. What was removed was the special code we had for > building with VS, but not the ability to build with VS. You can build with VS > using meson (see for example drongo on the buildfarm) Wow, okay, I am not surprised I was confused. New text is: <!-- Author: Michael Paquier <michael@paquier.xyz> 2023-12-20 [1301c80b2] Remove MSVC scripts --> <listitem> <para> Remove the Microsoft Visual Studio Studio-specific Postgres build option (Michael Paquier) </para> <para> Meson is now the only method for Visual Studio builds. </para> </listitem> <!-- Author: Michael Paquier <michael@paquier.xyz> 2023-12-20 [1301c80b2] Remove MSVC scripts --> <listitem> <para> Remove the Microsoft Visual Studio Studio-specific Postgres build option (Michael Paquier) </para> <para> Meson is now the only method for Visual Studio builds. </para> </listitem> -- Bruce Momjian <bruce@momjian.us> https://momjian.us EDB https://enterprisedb.com Only you can decide what is important to you.
On 2024-May-09, Bruce Momjian wrote: > However, I don't see it mentioned as a release note item in the commit > message or mentioned in our docs. I suppose the release note text would > be: > > Removing a PRIMARY KEY will remove the NOT NULL column specification > > Previously the NOT NULL specification would be retained. > > Do we have agreement that we want this release note item? Yes. Maybe we want some others too (especially regarding inheritance, but also regarding the way we handle the constraints internally), and maybe in this one we want different wording. How about something like this: Removing a primary key constraint may change the nullability characteristic of the columns that the primary key covered. If explicit not-null constraints exist on the same column, then they continue to be /known not nullable/; otherwise they become /possibly nullable/. This is largely based on the SQL standard's language of a column descriptor having a "nullability characteristic", which for columns with not-null or primary key constraints is "known not null". I don't think we use those terms anywhere. I hope this isn't too confusing. The standard's text on this, in section "4.13 Columns, fields, and attributes", is Every column has a nullability characteristic that indicates whether the value from that column can be the null value. A nullability characteristic is either known not nullable or possibly nullable. -- Álvaro Herrera 48°01'N 7°57'E — https://www.EnterpriseDB.com/
On Thu, May 9, 2024 at 4:04 PM Bruce Momjian <bruce@momjian.us> wrote: > I welcome feedback. For some reason it was an easier job than usual. > 2024-01-25 [820b5af73] jit: Require at least LLVM 10. > Require LLVM version 10 or later (Peter Eisentraut) Peter reviewed, I authored, and I think you intend to list authors in parentheses.
On Fri, May 10, 2024 at 08:05:43AM +1200, Thomas Munro wrote: > On Thu, May 9, 2024 at 4:04 PM Bruce Momjian <bruce@momjian.us> wrote: > > I welcome feedback. For some reason it was an easier job than usual. > > > 2024-01-25 [820b5af73] jit: Require at least LLVM 10. > > > Require LLVM version 10 or later (Peter Eisentraut) > > Peter reviewed, I authored, and I think you intend to list authors in > parentheses. Yes, my mistake, fixed. -- Bruce Momjian <bruce@momjian.us> https://momjian.us EDB https://enterprisedb.com Only you can decide what is important to you.
On Thu, May 9, 2024 at 08:40:00PM +0200, Álvaro Herrera wrote:
> On 2024-May-09, Bruce Momjian wrote:
>
> > However, I don't see it mentioned as a release note item in the commit
> > message or mentioned in our docs. I suppose the release note text would
> > be:
> >
> > Removing a PRIMARY KEY will remove the NOT NULL column specification
> >
> > Previously the NOT NULL specification would be retained.
> >
> > Do we have agreement that we want this release note item?
>
> Yes. Maybe we want some others too (especially regarding inheritance,
> but also regarding the way we handle the constraints internally), and
> maybe in this one we want different wording. How about something like
> this:
>
> Removing a primary key constraint may change the nullability
> characteristic of the columns that the primary key covered.
>
> If explicit not-null constraints exist on the same column, then they
> continue to be /known not nullable/; otherwise they become /possibly
> nullable/.
>
> This is largely based on the SQL standard's language of a column
> descriptor having a "nullability characteristic", which for columns with
> not-null or primary key constraints is "known not null". I don't think
> we use those terms anywhere. I hope this isn't too confusing.
Yes, it was confusing, partly because it is using wording we don't use,
and partly because it is talking about what can go into the column,
rather than the visible column restriction NOT NULL. I also think "may"
is too imprecise.
How about:
Removing a primary key will remove a column's NOT NULL constraint
if the constraint was added by the primary key
Previously such NOT NULL constraints would remain after a primary
key was removed. A column-level NOT NULL constraint would not be
emoved.
Here is the PG 16 output:
CREATE TABLE test ( x INT CONSTRAINT test_pkey PRIMARY KEY );
Table "public.test"
Column | Type | Collation | Nullable | Default
--------+---------+-----------+----------+---------
x | integer | | not null |
Indexes:
"test_pkey" PRIMARY KEY, btree (x)
CREATE TABLE test_with_not_null (x INT NOT NULL CONSTRAINT test_pkey_with_not_null PRIMARY KEY);
Table "public.test_with_not_null"
Column | Type | Collation | Nullable | Default
--------+---------+-----------+----------+---------
x | integer | | not null |
Indexes:
"test_pkey_with_not_null" PRIMARY KEY, btree (x)
ALTER TABLE test DROP CONSTRAINT test_pkey;
Table "public.test"
Column | Type | Collation | Nullable | Default
--------+---------+-----------+----------+---------
--> x | integer | | not null |
ALTER TABLE test_with_not_null DROP CONSTRAINT test_pkey_with_not_null;
Table "public.test_with_not_null"
Column | Type | Collation | Nullable | Default
--------+---------+-----------+----------+---------
--> x | integer | | not null |
Here is the output in PG 17:
CREATE TABLE test ( x INT CONSTRAINT test_pkey PRIMARY KEY );
Table "public.test"
Column | Type | Collation | Nullable | Default
--------+---------+-----------+----------+---------
x | integer | | not null |
Indexes:
"test_pkey" PRIMARY KEY, btree (x)
CREATE TABLE test_with_not_null (x INT NOT NULL CONSTRAINT test_pkey_with_not_null PRIMARY KEY);
Table "public.test_with_not_null"
Column | Type | Collation | Nullable | Default
--------+---------+-----------+----------+---------
x | integer | | not null |
Indexes:
"test_pkey_with_not_null" PRIMARY KEY, btree (x)
ALTER TABLE test DROP CONSTRAINT test_pkey;
Table "public.test"
Column | Type | Collation | Nullable | Default
--------+---------+-----------+----------+---------
--> x | integer | | |
ALTER TABLE test_with_not_null DROP CONSTRAINT test_pkey_with_not_null;
Table "public.test_with_not_null"
Column | Type | Collation | Nullable | Default
--------+---------+-----------+----------+---------
--> x | integer | | not null |
Notice that the table without a _column_ NOT NULL removes the NOT NULL
designation after removing the primary key only in PG 17.
--
Bruce Momjian <bruce@momjian.us> https://momjian.us
EDB https://enterprisedb.com
Only you can decide what is important to you.
On Thu, May 9, 2024 at 9:34 AM Bruce Momjian <bruce@momjian.us> wrote: > > I have committed the first draft of the PG 17 release notes; you can > see the results here: > > https://momjian.us/pgsql_docs/release-17.html > > It will be improved until the final release. The item count is 188, > which is similar to recent releases: > > release-10: 189 > release-11: 170 > release-12: 180 > release-13: 178 > release-14: 220 > release-15: 184 > release-16: 206 > release-17: 188 > > I welcome feedback. For some reason it was an easier job than usual. Thanks a lot for this work Bruce! It looks like commit https://git.postgresql.org/gitweb/?p=postgresql.git;a=commitdiff;h=91f2cae7a4e664e9c0472b364c7db29d755ab151 is missing from daft release notes. Just curious to know if it's intentional or a miss out. -- Bharath Rupireddy PostgreSQL Contributors Team RDS Open Source Databases Amazon Web Services: https://aws.amazon.com
On Fri, May 10, 2024 at 01:54:30PM +0530, Bharath Rupireddy wrote: > On Thu, May 9, 2024 at 9:34 AM Bruce Momjian <bruce@momjian.us> wrote: > > > > I have committed the first draft of the PG 17 release notes; you can > > see the results here: > > > > https://momjian.us/pgsql_docs/release-17.html > > > > It will be improved until the final release. The item count is 188, > > which is similar to recent releases: > > > > release-10: 189 > > release-11: 170 > > release-12: 180 > > release-13: 178 > > release-14: 220 > > release-15: 184 > > release-16: 206 > > release-17: 188 > > > > I welcome feedback. For some reason it was an easier job than usual. > > Thanks a lot for this work Bruce! It looks like commit > https://git.postgresql.org/gitweb/?p=postgresql.git;a=commitdiff;h=91f2cae7a4e664e9c0472b364c7db29d755ab151 > is missing from daft release notes. Just curious to know if it's > intentional or a miss out. I did not mention it because the commit didn't mention any performance benefit and it seemed more like an internal change than something people needed to know about. I could reword and merge it into this item, if you think I should: Improve performance of heavily-contended WAL writes (Bharath Rupireddy) -- Bruce Momjian <bruce@momjian.us> https://momjian.us EDB https://enterprisedb.com Only you can decide what is important to you.
Bruce Momjian wrote: > have committed the first draft of the PG 17 release notes; you can > see the results here: > > https://momjian.us/pgsql_docs/release-17.html In the psql items, I'd suggest mentioning https://git.postgresql.org/gitweb/?p=postgresql.git;a=commit;h=90f5178 For the short description, maybe something like that: - Improve FETCH_COUNT to work with all queries (Daniel Vérité) Previously, results would be fetched in chunks only for queries that start with the SELECT keyword. Best regards, -- Daniel Vérité https://postgresql.verite.pro/ Twitter: @DanielVerite
On Thu, 9 May 2024 at 06:04, Bruce Momjian <bruce@momjian.us> wrote: > > I have committed the first draft of the PG 17 release notes; you can > see the results here: > > https://momjian.us/pgsql_docs/release-17.html Great work! There are two commits that I think would benefit from being listed (but maybe they are already listed and I somehow missed them, or they are left out on purpose for some reason): - c4ab7da60617f020e8d75b1584d0754005d71830 - cafe1056558fe07cdc52b95205588fcd80870362
On Fri, May 10, 2024 at 06:29:11PM +0200, Daniel Verite wrote: > Bruce Momjian wrote: > > > have committed the first draft of the PG 17 release notes; you can > > see the results here: > > > > https://momjian.us/pgsql_docs/release-17.html > > In the psql items, I'd suggest mentioning > > https://git.postgresql.org/gitweb/?p=postgresql.git;a=commit;h=90f5178 > > For the short description, maybe something like that: > > - Improve FETCH_COUNT to work with all queries (Daniel Vérité) > Previously, results would be fetched in chunks only for queries > that start with the SELECT keyword. Agreed, patch attached and applied. -- Bruce Momjian <bruce@momjian.us> https://momjian.us EDB https://enterprisedb.com Only you can decide what is important to you.
Attachment
Mhd
Enviado desde Outlook para Android
From: Bruce Momjian <bruce@momjian.us>
Sent: Friday, May 10, 2024 4:47:04 PM
To: Daniel Verite <daniel@manitou-mail.org>
Cc: PostgreSQL-development <pgsql-hackers@postgresql.org>
Subject: Re: First draft of PG 17 release notes
Sent: Friday, May 10, 2024 4:47:04 PM
To: Daniel Verite <daniel@manitou-mail.org>
Cc: PostgreSQL-development <pgsql-hackers@postgresql.org>
Subject: Re: First draft of PG 17 release notes
On Fri, May 10, 2024 at 06:29:11PM +0200, Daniel Verite wrote:
> Bruce Momjian wrote:
>
> > have committed the first draft of the PG 17 release notes; you can
> > see the results here:
> >
> > https://momjian.us/pgsql_docs/release-17.html
>
> In the psql items, I'd suggest mentioning
>
> https://git.postgresql.org/gitweb/?p=postgresql.git;a=commit;h=90f5178
>
> For the short description, maybe something like that:
>
> - Improve FETCH_COUNT to work with all queries (Daniel Vérité)
> Previously, results would be fetched in chunks only for queries
> that start with the SELECT keyword.
Agreed, patch attached and applied.
--
Bruce Momjian <bruce@momjian.us> https://momjian.us
EDB https://enterprisedb.com
Only you can decide what is important to you.
> Bruce Momjian wrote:
>
> > have committed the first draft of the PG 17 release notes; you can
> > see the results here:
> >
> > https://momjian.us/pgsql_docs/release-17.html
>
> In the psql items, I'd suggest mentioning
>
> https://git.postgresql.org/gitweb/?p=postgresql.git;a=commit;h=90f5178
>
> For the short description, maybe something like that:
>
> - Improve FETCH_COUNT to work with all queries (Daniel Vérité)
> Previously, results would be fetched in chunks only for queries
> that start with the SELECT keyword.
Agreed, patch attached and applied.
--
Bruce Momjian <bruce@momjian.us> https://momjian.us
EDB https://enterprisedb.com
Only you can decide what is important to you.
On Fri, May 10, 2024 at 06:50:54PM +0200, Jelte Fennema-Nio wrote: > On Thu, 9 May 2024 at 06:04, Bruce Momjian <bruce@momjian.us> wrote: > > > > I have committed the first draft of the PG 17 release notes; you can > > see the results here: > > > > https://momjian.us/pgsql_docs/release-17.html > > Great work! > > There are two commits that I think would benefit from being listed > (but maybe they are already listed and I somehow missed them, or they > are left out on purpose for some reason): I looked at both of these. In both cases I didn't see why the user would need to know these changes were made: --------------------------------------------------------------------------- > - c4ab7da60617f020e8d75b1584d0754005d71830 commit c4ab7da6061 Author: David Rowley <drowley@postgresql.org> Date: Sun Apr 7 21:20:18 2024 +1200 Avoid needless large memcpys in libpq socket writing Until now, when calling pq_putmessage to write new data to a libpq socket, all writes are copied into a buffer and that buffer gets flushed when full to avoid having to perform small writes to the socket. There are cases where we must write large amounts of data to the socket, sometimes larger than the size of the buffer. In this case, it's wasteful to memcpy this data into the buffer and flush it out, instead, we can send it directly from the memory location that the data is already stored in. Here we adjust internal_putbytes() so that after having just flushed the buffer to the socket, if the remaining bytes to send is as big or bigger than the buffer size, we just send directly rather than needlessly copying into the PqSendBuffer buffer first. Examples of operations that write large amounts of data in one message are; outputting large tuples with SELECT or COPY TO STDOUT and pg_basebackup. Author: Melih Mutlu Reviewed-by: Heikki Linnakangas Reviewed-by: Jelte Fennema-Nio Reviewed-by: David Rowley Reviewed-by: Ranier Vilela Reviewed-by: Andres Freund Discussion: https://postgr.es/m/CAGPVpCR15nosj0f6xe-c2h477zFR88q12e6WjEoEZc8ZYkTh3Q@mail.gmail.com > - cafe1056558fe07cdc52b95205588fcd80870362 commit cafe1056558 Author: Robert Haas <rhaas@postgresql.org> Date: Tue Apr 2 10:26:10 2024 -0400 Allow SIGINT to cancel psql database reconnections. After installing the SIGINT handler in psql, SIGINT can no longer cancel database reconnections. For instance, if the user starts a reconnection and then needs to do some form of interaction (ie psql is polling), there is no way to cancel the reconnection process currently. Use PQconnectStartParams() in order to insert a cancel_pressed check into the polling loop. Tristan Partin, reviewed by Gurjeet Singh, Heikki Linnakangas, Jelte Fennema-Nio, and me. Discussion: http://postgr.es/m/D08WWCPVHKHN.3QELIKZJ2D9RZ@neon.tech -- Bruce Momjian <bruce@momjian.us> https://momjian.us EDB https://enterprisedb.com Only you can decide what is important to you.
Bruce Momjian <bruce@momjian.us> writes:
> On Fri, May 10, 2024 at 06:50:54PM +0200, Jelte Fennema-Nio wrote:
>> There are two commits that I think would benefit from being listed
>> (but maybe they are already listed and I somehow missed them, or they
>> are left out on purpose for some reason):
> I looked at both of these. In both cases I didn't see why the user
> would need to know these changes were made:
I agree that the buffering change is not likely interesting, but
the fact that you can now control-C out of a psql "\c" command
is user-visible. People might have internalized the fact that
it didn't work, or created complicated workarounds.
regards, tom lane
On Fri, May 10, 2024 at 05:31:33PM -0400, Tom Lane wrote: > Bruce Momjian <bruce@momjian.us> writes: > > On Fri, May 10, 2024 at 06:50:54PM +0200, Jelte Fennema-Nio wrote: > >> There are two commits that I think would benefit from being listed > >> (but maybe they are already listed and I somehow missed them, or they > >> are left out on purpose for some reason): > > > I looked at both of these. In both cases I didn't see why the user > > would need to know these changes were made: > > I agree that the buffering change is not likely interesting, but > the fact that you can now control-C out of a psql "\c" command > is user-visible. People might have internalized the fact that > it didn't work, or created complicated workarounds. It was not clear to me what the user-visible behavior was with the SIGINT control. Yes, based on your details, it should be mentioned. -- Bruce Momjian <bruce@momjian.us> https://momjian.us EDB https://enterprisedb.com Only you can decide what is important to you.
Hello Bruce, > I have committed the first draft of the PG 17 release notes; you can > see the results here: > > https://momjian.us/pgsql_docs/release-17.html Thank you for working on this! > I welcome feedback. For some reason it was an easier job than usual. Do you think we need to add the following 2 items? - 9f133763961e280d8ba692bcad0b061b861e9138 this is an optimizer transform improvement. - a8a968a8212ee3ef7f22795c834b33d871fac262 this is an optimizer costing improvement. Both of them can generate a better plan on some cases. -- Best Regards Andy Fan
On Sat, 11 May 2024 at 17:32, Andy Fan <zhihuifan1213@163.com> wrote: > Do you think we need to add the following 2 items? > > - 9f133763961e280d8ba692bcad0b061b861e9138 this is an optimizer > transform improvement. I think this should be in the release notes. Suggest: * Allow correlated IN subqueries to be transformed into joins (Andy Fan, Tom Lane) > - a8a968a8212ee3ef7f22795c834b33d871fac262 this is an optimizer costing > improvement. > > Both of them can generate a better plan on some cases. I think this should be present too. Suggest: * Improve optimizer's ability to use cheap startup plans when querying partitioned tables, inheritance parents and for UNION ALL (Andy Fan, David Rowley) Both under "E.1.3.1.1. Optimizer" David
On Fri, 10 May 2024 at 23:31, Tom Lane <tgl@sss.pgh.pa.us> wrote: > > Bruce Momjian <bruce@momjian.us> writes: > > I looked at both of these. In both cases I didn't see why the user > > would need to know these changes were made: > > I agree that the buffering change is not likely interesting, but > the fact that you can now control-C out of a psql "\c" command > is user-visible. People might have internalized the fact that > it didn't work, or created complicated workarounds. The buffering change improved performance up to ~40% in some of the benchmarks. The case it improves mostly is COPY of large rows and streaming a base backup. That sounds user-visible enough to me to warrant an entry imho.
On 5/11/24 09:57, Jelte Fennema-Nio wrote: > On Fri, 10 May 2024 at 23:31, Tom Lane <tgl@sss.pgh.pa.us> wrote: >> >> Bruce Momjian <bruce@momjian.us> writes: >> > I looked at both of these. In both cases I didn't see why the user >> > would need to know these changes were made: >> >> I agree that the buffering change is not likely interesting, but >> the fact that you can now control-C out of a psql "\c" command >> is user-visible. People might have internalized the fact that >> it didn't work, or created complicated workarounds. > > The buffering change improved performance up to ~40% in some of the > benchmarks. The case it improves mostly is COPY of large rows and > streaming a base backup. That sounds user-visible enough to me to > warrant an entry imho. +1 -- Joe Conway PostgreSQL Contributors Team RDS Open Source Databases Amazon Web Services: https://aws.amazon.com
On 2024-05-09 Th 00:03, Bruce Momjian wrote: > I have committed the first draft of the PG 17 release notes; you can > see the results here: > > https://momjian.us/pgsql_docs/release-17.html > > It will be improved until the final release. The item count is 188, > which is similar to recent releases: > > release-10: 189 > release-11: 170 > release-12: 180 > release-13: 178 > release-14: 220 > release-15: 184 > release-16: 206 > release-17: 188 > > I welcome feedback. For some reason it was an easier job than usual. I don't like blowing my own horn but I feel commit 3311ea86ed "Introduce a non-recursive JSON parser" should be in the release notes. This isn't something that's purely internal, but it could be used by an extension or a client program to parse JSON documents that are too large to handle with the existing API. Maybe "Introduce an incremental JSON parser" would have been a better headline. cheers andrew -- Andrew Dunstan EDB: https://www.enterprisedb.com
On Fri, May 10, 2024 at 7:20 PM Bruce Momjian <bruce@momjian.us> wrote: > > > Thanks a lot for this work Bruce! It looks like commit > > https://git.postgresql.org/gitweb/?p=postgresql.git;a=commitdiff;h=91f2cae7a4e664e9c0472b364c7db29d755ab151 > > is missing from daft release notes. Just curious to know if it's > > intentional or a miss out. > > I did not mention it because the commit didn't mention any performance > benefit and it seemed more like an internal change than something people > needed to know about. Yes, it's an internal feature for someone not using Direct IO for WAL and helps achieve things mentioned at https://www.postgresql.org/message-id/flat/20230125211540.zylu74dj2uuh3k7w%40awork3.anarazel.de#0cac0a0d219129e32329831adea05db5 (I'm hoping to target them for PG18). It starts to show visible benefits if someone enables direct IO for WAL (for whatever reasons) https://www.postgresql.org/message-id/CALj2ACV6rS%2B7iZx5%2BoAvyXJaN4AG-djAQeM1mrM%3DYSDkVrUs7g%40mail.gmail.com and https://www.postgresql.org/message-id/20230127061745.46yu4ksitzociwkt%40awork3.anarazel.de. I'm okay if 91f2cae7 is left out for the reason that Direct IO for WAL isn't something used in production and debug_io_direct is a developer option. > I could reword and merge it into this item, if > you think I should: > > Improve performance of heavily-contended WAL writes (Bharath Rupireddy) I think both the commits are for different purposes - one is for WAL wrties, another is for WAL reads. -- Bharath Rupireddy PostgreSQL Contributors Team RDS Open Source Databases Amazon Web Services: https://aws.amazon.com
On Fri, May 10, 2024 at 05:31:33PM -0400, Tom Lane wrote: > Bruce Momjian <bruce@momjian.us> writes: > > On Fri, May 10, 2024 at 06:50:54PM +0200, Jelte Fennema-Nio wrote: > >> There are two commits that I think would benefit from being listed > >> (but maybe they are already listed and I somehow missed them, or they > >> are left out on purpose for some reason): > > > I looked at both of these. In both cases I didn't see why the user > > would need to know these changes were made: > > I agree that the buffering change is not likely interesting, but > the fact that you can now control-C out of a psql "\c" command > is user-visible. People might have internalized the fact that > it didn't work, or created complicated workarounds. Agreed, attached patch applied. -- Bruce Momjian <bruce@momjian.us> https://momjian.us EDB https://enterprisedb.com Only you can decide what is important to you.
Attachment
On Sat, May 11, 2024 at 10:24:39AM -0400, Joe Conway wrote: > On 5/11/24 09:57, Jelte Fennema-Nio wrote: > > On Fri, 10 May 2024 at 23:31, Tom Lane <tgl@sss.pgh.pa.us> wrote: > > > > > > Bruce Momjian <bruce@momjian.us> writes: > > > > I looked at both of these. In both cases I didn't see why the user > > > > would need to know these changes were made: > > > > > > I agree that the buffering change is not likely interesting, but > > > the fact that you can now control-C out of a psql "\c" command > > > is user-visible. People might have internalized the fact that > > > it didn't work, or created complicated workarounds. > > > > The buffering change improved performance up to ~40% in some of the > > benchmarks. The case it improves mostly is COPY of large rows and > > streaming a base backup. That sounds user-visible enough to me to > > warrant an entry imho. > > +1 Attached patch applied. -- Bruce Momjian <bruce@momjian.us> https://momjian.us EDB https://enterprisedb.com Only you can decide what is important to you.
Attachment
On Sat, May 11, 2024 at 01:27:25PM +0800, Andy Fan wrote: > > Hello Bruce, > > > I have committed the first draft of the PG 17 release notes; you can > > see the results here: > > > > https://momjian.us/pgsql_docs/release-17.html > > Thank you for working on this! > > > I welcome feedback. For some reason it was an easier job than usual. > > Do you think we need to add the following 2 items? > > - 9f133763961e280d8ba692bcad0b061b861e9138 this is an optimizer > transform improvement. It was unclear from the commit message exactly what user-visible optimization this allowed. Do you have details? > - a8a968a8212ee3ef7f22795c834b33d871fac262 this is an optimizer costing > improvement. Does this allow faster UNION ALL with LIMIT, perhaps? -- Bruce Momjian <bruce@momjian.us> https://momjian.us EDB https://enterprisedb.com Only you can decide what is important to you.
jian he <jian.universality@gmail.com> 于2024年5月9日周四 18:00写道：
On Thu, May 9, 2024 at 12:04 PM Bruce Momjian <bruce@momjian.us> wrote:
>
> I have committed the first draft of the PG 17 release notes; you can
> see the results here:
>
> https://momjian.us/pgsql_docs/release-17.html
>
another potential incompatibilities issue:
ALTER TABLE DROP PRIMARY KEY
see:
https://www.postgresql.org/message-id/202404181849.6frtmajobe27%40alvherre.pgsql
Since Alvaro has reverted all changes to not-null constraints, so will not have potential incompatibilities issue.
-- Tender Wang
OpenPie: https://en.openpie.com/Bruce Momjian <bruce@momjian.us> writes: > On Sat, May 11, 2024 at 01:27:25PM +0800, Andy Fan wrote: >> >> Hello Bruce, >> >> > I have committed the first draft of the PG 17 release notes; you can >> > see the results here: >> > >> > https://momjian.us/pgsql_docs/release-17.html >> >> Thank you for working on this! >> >> > I welcome feedback. For some reason it was an easier job than usual. >> >> Do you think we need to add the following 2 items? >> >> - 9f133763961e280d8ba692bcad0b061b861e9138 this is an optimizer >> transform improvement. > > It was unclear from the commit message exactly what user-visible > optimization this allowed. Do you have details? Yes, It allows the query like "SELECT * FROM t1 WHERE t1.a in (SELECT a FROM t2 WHERE t2.b = t1.b)" be pulled up a semi join, hence more join methods / join orders are possible. > >> - a8a968a8212ee3ef7f22795c834b33d871fac262 this is an optimizer costing >> improvement. > > Does this allow faster UNION ALL with LIMIT, perhaps? Yes, for example: (subquery-1) UNION ALL (subquery-2) LIMIT n; When planning the subquery-1 or subquery-2, limit N should be considered. As a consequence, maybe hash join should be replaced with Nested Loop. Before this commits, it is ignored if it is flatten into appendrel, and the "flatten" happens very often. David provided a summary for the both commits in [1]. [1] https://www.postgresql.org/message-id/CAApHDvqAQgq27LgYmJ85VVGTR0%3DhRW6HHq2oZgK0ZiYC_a%2BEww%40mail.gmail.com -- Best Regards Andy Fan
Hi everybody, Being a technical writer, I attached a small patch that fixes minor language stuff. Thank you. Elena Indrupskaya Postgres Professional Company Moscow, Russia On 09.05.2024 07:03, Bruce Momjian wrote: > I have committed the first draft of the PG 17 release notes; you can > see the results here: > > https://momjian.us/pgsql_docs/release-17.html > > It will be improved until the final release. The item count is 188, > which is similar to recent releases: > > release-10: 189 > release-11: 170 > release-12: 180 > release-13: 178 > release-14: 220 > release-15: 184 > release-16: 206 > release-17: 188 > > I welcome feedback. For some reason it was an easier job than usual. >
Attachment
On Tue, 14 May 2024 at 02:56, Bruce Momjian <bruce@momjian.us> wrote: > > On Sat, May 11, 2024 at 10:24:39AM -0400, Joe Conway wrote: > > On 5/11/24 09:57, Jelte Fennema-Nio wrote: > > > The buffering change improved performance up to ~40% in some of the > > > benchmarks. The case it improves mostly is COPY of large rows and > > > streaming a base backup. That sounds user-visible enough to me to > > > warrant an entry imho. > > > > +1 > > Attached patch applied. I think we shouldn't list this under the libpq changes and shouldn't mention libpq in the description, since this patch changes src/backend/libpq files instead of src/interfaces/libpq files. I think it should be in the "General performance" section and describe the change as something like the below: Improve performance when transferring large blocks of data to a client PS. I completely understand that this was not clear from the commit message.
On Thu, May 9, 2024 at 5:03 AM Bruce Momjian <bruce@momjian.us> wrote > > > I welcome feedback. For some reason it was an easier job than usual. This looks better if "more case" -> "more cases" : > Allow query nodes to be run in parallel in more case (Tom Lane)
On Thu, May 9, 2024 at 12:04 AM Bruce Momjian <bruce@momjian.us> wrote: > > I have committed the first draft of the PG 17 release notes; you can > see the results here: > > https://momjian.us/pgsql_docs/release-17.html I had two comments: -------- I think the read stream item: "Allow the grouping of file system reads with the new system variable io_combine_limit" Might be better if it mentions the effect, like: "Reduce system calls by automatically merging reads up to io_combine_limit" ------- For the vacuum feature: "Allow vacuum to more efficiently remove and freeze tuples" I think that we need to more clearly point out the implications of the feature added by Sawada-san (and reviewed by John) in 667e65aac35497. Vacuum no longer uses a fixed amount of memory for dead tuple TID storage and it is not preallocated. This affects users as they may want to change their configuration (and expectations). If you make that item more specific to their work, you should also remove my name, as the work I did on vacuum this release was unrelated to their work on dead tuple TID storage. The work Heikki and I did which culminated in 6dbb490261 mainly has the impact of improving vacuum's performance (vacuum emits less WAL and is more efficient). So you could argue for removing it from the release notes if you are using the requirement that performance improvements don't go in the release notes. However, one of the preliminary commits for this f83d70976 does change WAL format. There are three WAL records which no longer exist as separate records. Do users care about this? - Melanie
On Tue, May 14, 2024 at 10:32:14AM +0800, Andy Fan wrote: > Bruce Momjian <bruce@momjian.us> writes: > > It was unclear from the commit message exactly what user-visible > > optimization this allowed. Do you have details? > > Yes, It allows the query like "SELECT * FROM t1 WHERE t1.a in (SELECT a > FROM t2 WHERE t2.b = t1.b)" be pulled up a semi join, hence more join > methods / join orders are possible. > > > Yes, for example: (subquery-1) UNION ALL (subquery-2) LIMIT n; > > When planning the subquery-1 or subquery-2, limit N should be > considered. As a consequence, maybe hash join should be replaced with > Nested Loop. Before this commits, it is ignored if it is flatten into > appendrel, and the "flatten" happens very often. > > David provided a summary for the both commits in [1]. Okay, attached patch applied. -- Bruce Momjian <bruce@momjian.us> https://momjian.us EDB https://enterprisedb.com Only you can decide what is important to you.
Attachment
On Sat, May 11, 2024 at 03:32:55PM -0400, Andrew Dunstan wrote: > > On 2024-05-09 Th 00:03, Bruce Momjian wrote: > > I have committed the first draft of the PG 17 release notes; you can > > see the results here: > > > > https://momjian.us/pgsql_docs/release-17.html > > > > It will be improved until the final release. The item count is 188, > > which is similar to recent releases: > > > > release-10: 189 > > release-11: 170 > > release-12: 180 > > release-13: 178 > > release-14: 220 > > release-15: 184 > > release-16: 206 > > release-17: 188 > > > > I welcome feedback. For some reason it was an easier job than usual. > > > I don't like blowing my own horn but I feel commit 3311ea86ed "Introduce a > non-recursive JSON parser" should be in the release notes. This isn't > something that's purely internal, but it could be used by an extension or a > client program to parse JSON documents that are too large to handle with the > existing API. > > Maybe "Introduce an incremental JSON parser" would have been a better > headline. Well, this gets into a level of detail that is beyond the average reader. I think at that level people will need to read the git logs or review the code. Do we use it for anything yet? It could be put in the source code section but I try to only have user-visible stuff there. -- Bruce Momjian <bruce@momjian.us> https://momjian.us EDB https://enterprisedb.com Only you can decide what is important to you.
On Tue, May 14, 2024 at 01:34:56PM +0300, Elena Indrupskaya wrote: > Being a technical writer, I attached a small patch that fixes minor language > stuff. You are absolutely correct. Patch applied, thanks. -- Bruce Momjian <bruce@momjian.us> https://momjian.us EDB https://enterprisedb.com Only you can decide what is important to you.
On Tue, May 14, 2024 at 02:20:24PM +0200, Jelte Fennema-Nio wrote: > On Tue, 14 May 2024 at 02:56, Bruce Momjian <bruce@momjian.us> wrote: > > > > On Sat, May 11, 2024 at 10:24:39AM -0400, Joe Conway wrote: > > > On 5/11/24 09:57, Jelte Fennema-Nio wrote: > > > > The buffering change improved performance up to ~40% in some of the > > > > benchmarks. The case it improves mostly is COPY of large rows and > > > > streaming a base backup. That sounds user-visible enough to me to > > > > warrant an entry imho. > > > > > > +1 > > > > Attached patch applied. > > I think we shouldn't list this under the libpq changes and shouldn't > mention libpq in the description, since this patch changes > src/backend/libpq files instead of src/interfaces/libpq files. I think > it should be in the "General performance" section and describe the > change as something like the below: > > Improve performance when transferring large blocks of data to a client > > PS. I completely understand that this was not clear from the commit message. Okay, I went with your wording. Attached patch applied. -- Bruce Momjian <bruce@momjian.us> https://momjian.us EDB https://enterprisedb.com Only you can decide what is important to you.
Attachment
On Tue, May 14, 2024 at 02:58:41PM +0100, Pantelis Theodosiou wrote: > On Thu, May 9, 2024 at 5:03 AM Bruce Momjian <bruce@momjian.us> wrote > > > > > > I welcome feedback. For some reason it was an easier job than usual. > > This looks better if "more case" -> "more cases" : > > Allow query nodes to be run in parallel in more case (Tom Lane) Yes, you are correct, fixed. -- Bruce Momjian <bruce@momjian.us> https://momjian.us EDB https://enterprisedb.com Only you can decide what is important to you.
On Tue, May 14, 2024 at 03:39:26PM -0400, Melanie Plageman wrote: > On Thu, May 9, 2024 at 12:04 AM Bruce Momjian <bruce@momjian.us> wrote: > > > > I have committed the first draft of the PG 17 release notes; you can > > see the results here: > > > > https://momjian.us/pgsql_docs/release-17.html > > I had two comments: > > -------- > I think the read stream item: > > "Allow the grouping of file system reads with the new system variable > io_combine_limit" > > Might be better if it mentions the effect, like: > > "Reduce system calls by automatically merging reads up to io_combine_limit" Uh, as I understand it, the reduced number of system calls is not the value of the feature, but rather the ability to request a larger block from the I/O subsystem. Without it, you have to make a request and wait for each request to finish. I am open to new wording, but I am not sure your new wording is accurate. > ------- > For the vacuum feature: > > "Allow vacuum to more efficiently remove and freeze tuples" > > I think that we need to more clearly point out the implications of the > feature added by Sawada-san (and reviewed by John) in 667e65aac35497. > Vacuum no longer uses a fixed amount of memory for dead tuple TID > storage and it is not preallocated. This affects users as they may > want to change their configuration (and expectations). > > If you make that item more specific to their work, you should also > remove my name, as the work I did on vacuum this release was unrelated > to their work on dead tuple TID storage. > > The work Heikki and I did which culminated in 6dbb490261 mainly has > the impact of improving vacuum's performance (vacuum emits less WAL > and is more efficient). So you could argue for removing it from the > release notes if you are using the requirement that performance > improvements don't go in the release notes. > > However, one of the preliminary commits for this f83d70976 does change > WAL format. There are three WAL records which no longer exist as > separate records. Do users care about this? I don't think users really care about these details, just that it is faster so they will not be surprised if there is a change. I was purposely vague to group multiple commits into the single item. By grouping them together, I got enough impact to warrant listing it. If you split it apart, it is harder to justify mentioning them. -- Bruce Momjian <bruce@momjian.us> https://momjian.us EDB https://enterprisedb.com Only you can decide what is important to you.
On Thu, May 9, 2024 at 10:48 PM Bruce Momjian <bruce@momjian.us> wrote: > > On Thu, May 9, 2024 at 02:17:12PM +0900, Masahiko Sawada wrote: > > Hi, > > > > > Also, please consider the following item: > > > > - Improve eviction algorithm in ReorderBuffer using max-heap for many > > subtransactions (5bec1d6bc) > > I looked at that item and I don't have a generic "make logical > replication apply faster" item to merge it into, and many > subtransactions seemed like enough of an edge-case that I didn't think > mentioning it make sense. Can you see a good place to add it? I think that since many subtransactions cases are no longer becoming edge-cases these days, we needed to improve that and it might be helpful for users to mention it. How about the following item for example? Improve logical decoding performance in cases where there are many subtransactions. > > > Finally, should we mention the following commit in the release note? > > It's not a user-visible change but added a new regression test module. > > > > - Add tests for XID wraparound (e255b646a) > > I don't normally add testing infrastructure changes unless they are > major. I've seen we had such item, for example in PG14 release note: Add a test module for the regular expression package (Tom Lane) But if our policy has already changed, I'm okay with not mentioning the xid_wraparound test in the PG17 release note. Regards, -- Masahiko Sawada Amazon Web Services: https://aws.amazon.com
On Tue, May 14, 2024 at 10:22:35AM +0800, Tender Wang wrote: > > > jian he <jian.universality@gmail.com> 于2024年5月9日周四 18:00写道： > > On Thu, May 9, 2024 at 12:04 PM Bruce Momjian <bruce@momjian.us> wrote: > > > > I have committed the first draft of the PG 17 release notes; you can > > see the results here: > > > > https://momjian.us/pgsql_docs/release-17.html > > > > another potential incompatibilities issue: > ALTER TABLE DROP PRIMARY KEY > > see: > https://www.postgresql.org/message-id/ > 202404181849.6frtmajobe27%40alvherre.pgsql > > > > Since Alvaro has reverted all changes to not-null constraints, so will not have > potential incompatibilities issue. Agreed. -- Bruce Momjian <bruce@momjian.us> https://momjian.us EDB https://enterprisedb.com Only you can decide what is important to you.
On Wed, 15 May 2024 at 13:00, Bruce Momjian <bruce@momjian.us> wrote: > > On Tue, May 14, 2024 at 03:39:26PM -0400, Melanie Plageman wrote: > > "Reduce system calls by automatically merging reads up to io_combine_limit" > > Uh, as I understand it, the reduced number of system calls is not the > value of the feature, but rather the ability to request a larger block > from the I/O subsystem. Without it, you have to make a request and wait > for each request to finish. I am open to new wording, but I am not sure > your new wording is accurate. I think you have the cause and effect backwards. There's no advantage to reading 128KB if you only need 8KB. It's the fact that doing *larger* reads allows *fewer* reads that allows it to be more efficient. There are also the efficiency gains from fadvise POSIX_FADV_WILLNEED. I'm unsure how to jam that into a short sentence. Maybe; "Optimize reading of tables by allowing pages to be prefetched and read in chunks up to io_combine_limit", or a bit more buzzy; "Optimize reading of tables by allowing pages to be prefetched and performing vectored reads in chunks up to io_combine_limit". David
On Wed, May 15, 2024 at 02:03:32PM +1200, David Rowley wrote: > On Wed, 15 May 2024 at 13:00, Bruce Momjian <bruce@momjian.us> wrote: > > > > On Tue, May 14, 2024 at 03:39:26PM -0400, Melanie Plageman wrote: > > > "Reduce system calls by automatically merging reads up to io_combine_limit" > > > > Uh, as I understand it, the reduced number of system calls is not the > > value of the feature, but rather the ability to request a larger block > > from the I/O subsystem. Without it, you have to make a request and wait > > for each request to finish. I am open to new wording, but I am not sure > > your new wording is accurate. > > I think you have the cause and effect backwards. There's no advantage > to reading 128KB if you only need 8KB. It's the fact that doing > *larger* reads allows *fewer* reads that allows it to be more > efficient. There are also the efficiency gains from fadvise > POSIX_FADV_WILLNEED. I'm unsure how to jam that into a short sentence. > Maybe; "Optimize reading of tables by allowing pages to be prefetched > and read in chunks up to io_combine_limit", or a bit more buzzy; > "Optimize reading of tables by allowing pages to be prefetched and > performing vectored reads in chunks up to io_combine_limit". Yes, my point is that it is not the number of system calls or system call overhead that is the advantage of this patch, but the ability to request more of the I/O system in one call, which is not the same as system calls. I can use your wording, but how much prefetching to we enable now? -- Bruce Momjian <bruce@momjian.us> https://momjian.us EDB https://enterprisedb.com Only you can decide what is important to you.
On Wed, May 15, 2024 at 10:10:28AM +0900, Masahiko Sawada wrote: > > I looked at that item and I don't have a generic "make logical > > replication apply faster" item to merge it into, and many > > subtransactions seemed like enough of an edge-case that I didn't think > > mentioning it make sense. Can you see a good place to add it? > > I think that since many subtransactions cases are no longer becoming > edge-cases these days, we needed to improve that and it might be > helpful for users to mention it. How about the following item for > example? > > Improve logical decoding performance in cases where there are many > subtransactions. Okay, item added in the attached applied patch. > > > Finally, should we mention the following commit in the release note? > > > It's not a user-visible change but added a new regression test module. > > > > > > - Add tests for XID wraparound (e255b646a) > > > > I don't normally add testing infrastructure changes unless they are > > major. > > I've seen we had such item, for example in PG14 release note: > > Add a test module for the regular expression package (Tom Lane) > > But if our policy has already changed, I'm okay with not mentioning > the xid_wraparound test in the PG17 release note. Uh, that PG 14 test suite was huge and flushed out a lot of bugs, not only in our regex code but I think in the TCL/Henry Spencer regex library we inherited. We add 10-40 tests every year, and how many do I mention in the release notes? You had to go back to PG 14 to find one. We have not changed our release note "test item" criteria --- I only mention tests that are significant to our userbase. I think that test suite was significant to anyone using the TCL/Henry Spencer regex library. If you want your test mentioned, you have to explain why it is useful for users to know about it, or the value it brings them. -- Bruce Momjian <bruce@momjian.us> https://momjian.us EDB https://enterprisedb.com Only you can decide what is important to you.
Attachment
On Wed, 15 May 2024 at 14:06, Bruce Momjian <bruce@momjian.us> wrote: > I can use your wording, but how much prefetching to we enable now? I believe the read stream API is used for Seq Scan, ANALYZE and pg_prewarm(). fadvise() is used when the next buffer that's required is not in shared buffers on any build that has defined HAVE_DECL_POSIX_FADVISE. David
On 2024-May-14, Bruce Momjian wrote: > On Tue, May 14, 2024 at 03:39:26PM -0400, Melanie Plageman wrote: > > I think that we need to more clearly point out the implications of the > > feature added by Sawada-san (and reviewed by John) in 667e65aac35497. > > Vacuum no longer uses a fixed amount of memory for dead tuple TID > > storage and it is not preallocated. This affects users as they may > > want to change their configuration (and expectations). > > > > If you make that item more specific to their work, you should also > > remove my name, as the work I did on vacuum this release was unrelated > > to their work on dead tuple TID storage. > > > > The work Heikki and I did which culminated in 6dbb490261 mainly has > > the impact of improving vacuum's performance (vacuum emits less WAL > > and is more efficient). So you could argue for removing it from the > > release notes if you are using the requirement that performance > > improvements don't go in the release notes. > > I don't think users really care about these details, just that it is > faster so they will not be surprised if there is a change. I was > purposely vague to group multiple commits into the single item. By > grouping them together, I got enough impact to warrant listing it. If > you split it apart, it is harder to justify mentioning them. I disagree with this. IMO the impact of the Sawada/Naylor change is likely to be enormous for people with large tables and large numbers of tuples to clean up (I know we've had a number of customers in this situation, I can't imagine any Postgres service provider that doesn't). The fact that maintenance_work_mem is no longer capped at 1GB is very important and I think we should mention that explicitly in the release notes, as setting it higher could make a big difference in vacuum run times. I don't know what's the impact of the Plageman/Linnakangas work, but since there are no user-visible consequences other than it being faster, I agree it could be put more succintly, perhaps together as a sub-para of the same item. What about something like this? <para> Lift the 1 GB allocation limit for vacuum memory usage for dead tuples, and make storage more compact and performant. </para> <para> This can reduce the number of index passes that vacuum has to perform for tables with many dead tuples, shortening vacuum times. </para> <para> Also, the WAL traffic caused by vacuum has been made more compact. </para> > > However, one of the preliminary commits for this f83d70976 does > > change WAL format. There are three WAL records which no longer exist > > as separate records. Do users care about this? I don't think so. -- Álvaro Herrera 48°01'N 7°57'E — https://www.EnterpriseDB.com/ "You don't solve a bad join with SELECT DISTINCT" #CupsOfFail https://twitter.com/connor_mc_d/status/1431240081726115845
On Wed, 15 May 2024 at 20:38, Alvaro Herrera <alvherre@alvh.no-ip.org> wrote: > > On 2024-May-14, Bruce Momjian wrote: > > I don't think users really care about these details, just that it is > > faster so they will not be surprised if there is a change. I was > > purposely vague to group multiple commits into the single item. By > > grouping them together, I got enough impact to warrant listing it. If > > you split it apart, it is harder to justify mentioning them. > > I disagree with this. IMO the impact of the Sawada/Naylor change is > likely to be enormous for people with large tables and large numbers of > tuples to clean up (I know we've had a number of customers in this > situation, I can't imagine any Postgres service provider that doesn't). > The fact that maintenance_work_mem is no longer capped at 1GB is very > important and I think we should mention that explicitly in the release > notes, as setting it higher could make a big difference in vacuum run > times. I very much agree with Alvaro here. IMO, this should be on the highlight feature list for v17. Prior to this, having to do multiple index scans because of filling maintenance_work_mem was a performance tragedy. If there were enough dead tuples to have filled maintenance_work_mem, then the indexes are large. Having to scan multiple large indexes multiple times isn't good use of I/O and CPU. As far as I understand it, this work means it'll be unlikely that a well-configured server will ever have to do multiple index passes. I don't think "enormous impact" is an exaggeration here. David
On Wed, May 15, 2024 at 4:38 AM Alvaro Herrera <alvherre@alvh.no-ip.org> wrote: > > On 2024-May-14, Bruce Momjian wrote: > > > On Tue, May 14, 2024 at 03:39:26PM -0400, Melanie Plageman wrote: > > > > I think that we need to more clearly point out the implications of the > > > feature added by Sawada-san (and reviewed by John) in 667e65aac35497. > > > Vacuum no longer uses a fixed amount of memory for dead tuple TID > > > storage and it is not preallocated. This affects users as they may > > > want to change their configuration (and expectations). > > > > > > If you make that item more specific to their work, you should also > > > remove my name, as the work I did on vacuum this release was unrelated > > > to their work on dead tuple TID storage. > > > > > > The work Heikki and I did which culminated in 6dbb490261 mainly has > > > the impact of improving vacuum's performance (vacuum emits less WAL > > > and is more efficient). So you could argue for removing it from the > > > release notes if you are using the requirement that performance > > > improvements don't go in the release notes. > > > > I don't think users really care about these details, just that it is > > faster so they will not be surprised if there is a change. I was > > purposely vague to group multiple commits into the single item. By > > grouping them together, I got enough impact to warrant listing it. If > > you split it apart, it is harder to justify mentioning them. > > I disagree with this. IMO the impact of the Sawada/Naylor change is > likely to be enormous for people with large tables and large numbers of > tuples to clean up (I know we've had a number of customers in this > situation, I can't imagine any Postgres service provider that doesn't). > The fact that maintenance_work_mem is no longer capped at 1GB is very > important and I think we should mention that explicitly in the release > notes, as setting it higher could make a big difference in vacuum run > times. > > I don't know what's the impact of the Plageman/Linnakangas work, but > since there are no user-visible consequences other than it being faster, > I agree it could be put more succintly, perhaps together as a sub-para > of the same item. > > What about something like this? > > <para> > Lift the 1 GB allocation limit for vacuum memory usage for dead > tuples, and make storage more compact and performant. > </para> > <para> > This can reduce the number of index passes that vacuum has to perform > for tables with many dead tuples, shortening vacuum times. > </para> > <para> > Also, the WAL traffic caused by vacuum has been made more compact. > </para> I think this wording and organization makes sense. I hadn't thought of using "traffic" to describe this, but I like it. Also +1 on the Sawada/Naylor change being on the highlight section of the release (as David suggested upthread). - Melanie
On Wed, May 15, 2024 at 7:36 AM Bruce Momjian <bruce@momjian.us> wrote: > > On Wed, May 15, 2024 at 02:03:32PM +1200, David Rowley wrote: > > On Wed, 15 May 2024 at 13:00, Bruce Momjian <bruce@momjian.us> wrote: > > > > > > On Tue, May 14, 2024 at 03:39:26PM -0400, Melanie Plageman wrote: > > > > "Reduce system calls by automatically merging reads up to io_combine_limit" > > > > > > Uh, as I understand it, the reduced number of system calls is not the > > > value of the feature, but rather the ability to request a larger block > > > from the I/O subsystem. Without it, you have to make a request and wait > > > for each request to finish. I am open to new wording, but I am not sure > > > your new wording is accurate. > > > > I think you have the cause and effect backwards. There's no advantage > > to reading 128KB if you only need 8KB. It's the fact that doing > > *larger* reads allows *fewer* reads that allows it to be more > > efficient. There are also the efficiency gains from fadvise > > POSIX_FADV_WILLNEED. I'm unsure how to jam that into a short sentence. > > Maybe; "Optimize reading of tables by allowing pages to be prefetched > > and read in chunks up to io_combine_limit", or a bit more buzzy; > > "Optimize reading of tables by allowing pages to be prefetched and > > performing vectored reads in chunks up to io_combine_limit". > > Yes, my point is that it is not the number of system calls or system > call overhead that is the advantage of this patch, but the ability to > request more of the I/O system in one call, which is not the same as > system calls. > > I can use your wording, but how much prefetching to we enable now? > Shouldn't we need to include commit b5a9b18cd0bc6f0124664999b31a00a264d16913 with this item? -- With Regards, Amit Kapila.
On Thu, May 9, 2024 at 12:04 PM Bruce Momjian <bruce@momjian.us> wrote: > > I have committed the first draft of the PG 17 release notes; you can > see the results here: > > https://momjian.us/pgsql_docs/release-17.html > >> Add local I/O block read/write timing statistics columns of pg_stat_statement (Nazir Bilal Yavuz) >> The new columns are "local_blk_read_time" and "local_blk_write_time". here, "pg_stat_statement" should be "pg_stat_statements"? >> Add optional fourth parameter to pg_stat_statements_reset() to allow for the resetting of only min/max statistics (AndreiZubkov) >> This parameter defaults to "false". here, "parameter", should be "argument"? maybe >> Add optional fourth boolean argument (minmax_only) to pg_stat_statements_reset() to allow for the resetting of only min/maxstatistics (Andrei Zubkov) >> This argument defaults to "false". ---------------------------------------------------------------- in section: E.1.2. Migration to Version 17 >> Rename I/O block read/write timing statistics columns of pg_stat_statement (Nazir Bilal Yavuz) >> This renames "blk_read_time" to "shared_blk_read_time", and "blk_write_time" to "shared_blk_write_time". "pg_stat_statement" should be "pg_stat_statements"? also, we only mentioned, pg_stat_statements some columns name changed in "E.1.2. Migration to Version 17" but if you look at the release note pg_stat_statements section, we added a bunch of columns, which are far more incompatibile than just colunm name changes. not sure we need add these in section "E.1.2. Migration to Version 17"
On Wed, May 15, 2024 at 09:13:14AM -0400, Melanie Plageman wrote: > I think this wording and organization makes sense. I hadn't thought of > using "traffic" to describe this, but I like it. > > Also +1 on the Sawada/Naylor change being on the highlight section of > the release (as David suggested upthread). Agreed, I went with the attached applied patch. -- Bruce Momjian <bruce@momjian.us> https://momjian.us EDB https://enterprisedb.com Only you can decide what is important to you.
Attachment
On Wednesday, May 15, 2024, jian he <jian.universality@gmail.com> wrote:
On Thu, May 9, 2024 at 12:04 PM Bruce Momjian <bruce@momjian.us> wrote:
>
> I have committed the first draft of the PG 17 release notes; you can
> see the results here:
>
> https://momjian.us/pgsql_docs/release-17.html
>
in section: E.1.2. Migration to Version 17
>> Rename I/O block read/write timing statistics columns of pg_stat_statement (Nazir Bilal Yavuz)
>> This renames "blk_read_time" to "shared_blk_read_time", and "blk_write_time" to "shared_blk_write_time".
we only mentioned, pg_stat_statements some columns name changed
in "E.1.2. Migration to Version 17"
but if you look at the release note pg_stat_statements section,
we added a bunch of columns, which are far more incompatibile than
just colunm name changes.
not sure we need add these in section "E.1.2. Migration to Version 17"
New columns are not a migration issue since nothing being migrated forward ever referenced them. Its the ones that existing code knows about that we’ve removed (including renames) that matter from a migration perspective.
David J.
On Thu, May 16, 2024 at 10:39:18AM +0800, jian he wrote: > On Thu, May 9, 2024 at 12:04 PM Bruce Momjian <bruce@momjian.us> wrote: > > > > I have committed the first draft of the PG 17 release notes; you can > > see the results here: > > > > https://momjian.us/pgsql_docs/release-17.html > > > > >> Add local I/O block read/write timing statistics columns of pg_stat_statement (Nazir Bilal Yavuz) > >> The new columns are "local_blk_read_time" and "local_blk_write_time". > here, "pg_stat_statement" should be "pg_stat_statements"? Agreed. > >> Add optional fourth parameter to pg_stat_statements_reset() to allow for the resetting of only min/max statistics (AndreiZubkov) > >> This parameter defaults to "false". > here, "parameter", should be "argument"? > > maybe > >> Add optional fourth boolean argument (minmax_only) to pg_stat_statements_reset() to allow for the resetting of onlymin/max statistics (Andrei Zubkov) > >> This argument defaults to "false". Sure. > ---------------------------------------------------------------- > in section: E.1.2. Migration to Version 17 > > >> Rename I/O block read/write timing statistics columns of pg_stat_statement (Nazir Bilal Yavuz) > >> This renames "blk_read_time" to "shared_blk_read_time", and "blk_write_time" to "shared_blk_write_time". > > "pg_stat_statement" should be "pg_stat_statements"? Yes, fixed. > also, we only mentioned, pg_stat_statements some columns name changed > in "E.1.2. Migration to Version 17" > but if you look at the release note pg_stat_statements section, > we added a bunch of columns, which are far more incompatibile than > just colunm name changes. > > not sure we need add these in section "E.1.2. Migration to Version 17" Well, new columns don't cause breakage like renamed columns, which is why I only put renames/removed columns in the migration section. Also, thanks everyone for the release notes feedback. In some cases I made a mistake, and in some cases I misjudged the item. -- Bruce Momjian <bruce@momjian.us> https://momjian.us EDB https://enterprisedb.com Only you can decide what is important to you.
On Thu, 16 May 2024 at 14:48, Bruce Momjian <bruce@momjian.us> wrote: > > On Wed, May 15, 2024 at 09:13:14AM -0400, Melanie Plageman wrote: > > Also +1 on the Sawada/Naylor change being on the highlight section of > > the release (as David suggested upthread). > > Agreed, I went with the attached applied patch. +Allow vacuum to more efficiently store tuple references and remove its memory limit (Masahiko Sawada, John Naylor) +</para> I don't want it to seem like I'm splitting hairs, but I'd drop the " and remove its memory limit" +<para> +Specifically, maintenance_work_mem and autovacuum_work_mem can now be configured to use more than one gigabyte of memory. WAL traffic caused by vacuum is also more compact. I'd say the first sentence above should be written as: "Additionally, vacuum no longer silently imposes a 1GB tuple reference limit even when maintenance_work_mem or autovacuum_work_mem are set to higher values" It's not "Specifically" as the "more efficiently store tuple references" isn't the same thing as removing the 1GB cap. Also, there was never a restriction in configuring maintenance_work_mem or autovacuum_work_mem to values higher than 1GB. The restriction was that vacuum was unable to utilize anything more than that. David
Hi, On 2024-05-15 10:38:20 +0200, Alvaro Herrera wrote: > I disagree with this. IMO the impact of the Sawada/Naylor change is > likely to be enormous for people with large tables and large numbers of > tuples to clean up (I know we've had a number of customers in this > situation, I can't imagine any Postgres service provider that doesn't). > The fact that maintenance_work_mem is no longer capped at 1GB is very > important and I think we should mention that explicitly in the release > notes, as setting it higher could make a big difference in vacuum run > times. +many. We're having this debate every release. I think the ongoing reticence to note performance improvements in the release notes is hurting Postgres. For one, performance improvements are one of the prime reason users upgrade. Without them being noted anywhere more dense than the commit log, it's very hard to figure out what improved for users. A halfway widely applicable performance improvement is far more impactful than many of the feature changes we do list in the release notes. For another, it's also very frustrating for developers that focus on performance. The reticence to note their work, while noting other, far smaller, things in the release notes, pretty much tells us that our work isn't valued. Greetings, Andres Freund
On Thu, May 9, 2024 at 12:04 PM Bruce Momjian <bruce@momjian.us> wrote: > > I have committed the first draft of the PG 17 release notes; you can > see the results here: > > https://momjian.us/pgsql_docs/release-17.html > >> Add jsonpath methods to convert JSON values to different data types (Jeevan Chalke) >> The jsonpath methods are .bigint(), .boolean(), .date(), .decimal([precision [, scale]]), .integer(), .number(), .string(),.time(), .time_tz(), .timestamp(), and .timestamp_tz(). I think it's slightly incorrect. for example: select jsonb_path_query('"2023-08-15"', '$.date()'); I think it does is trying to validate json value "2023-08-15" can be a date value, if so, print json string out, else error out. "convert JSON values to different data types" meaning that we are converting json values to another data type, date?
On Thu, 16 May 2024 at 05:48, Andres Freund <andres@anarazel.de> wrote: > We're having this debate every release. I think the ongoing reticence to note > performance improvements in the release notes is hurting Postgres. > > For one, performance improvements are one of the prime reason users > upgrade. Without them being noted anywhere more dense than the commit log, > it's very hard to figure out what improved for users. A halfway widely > applicable performance improvement is far more impactful than many of the > feature changes we do list in the release notes. > > For another, it's also very frustrating for developers that focus on > performance. The reticence to note their work, while noting other, far > smaller, things in the release notes, pretty much tells us that our work isn't > valued. +1 to the general gist of listing every perf improvement **and memory usage reduction** in the release notes. Most of them are already grouped together in a dedicated "General performance" section anyway, having that section be big would only be good imho to show that we're committed to improving perf. I think one thing would make this a lot easier though is if commits that knowlingy impact perf would clearly say so in the commit message, because now it's sometimes hard to spot as someone not deeply involved with the specific patch. e.g. c4ab7da606 doesn't mention performance at all, so I'm not surprised it wasn't listed initially. And while 667e65aac3 states that multiple rounds of heap scanning is now extremely rare, it doesn't explicitly state what the kind of perf impact can be expected because of that. Maybe something like introducing a common "Perf-Improvement: true" marker in the commit message and when doing so add a clear paragraph explaining the expected perf impact perf impact. Another option could be to add a "User Impact" section to the commit message, where an author could add their suggestion for a release note entry. So basically this suggestion boils down to more clearly mentioning user impact in commit messages, instead of mostly/only including technical/implementation details.
On 5/15/24 23:48, Andres Freund wrote: > On 2024-05-15 10:38:20 +0200, Alvaro Herrera wrote: >> I disagree with this. IMO the impact of the Sawada/Naylor change is >> likely to be enormous for people with large tables and large numbers of >> tuples to clean up (I know we've had a number of customers in this >> situation, I can't imagine any Postgres service provider that doesn't). >> The fact that maintenance_work_mem is no longer capped at 1GB is very >> important and I think we should mention that explicitly in the release >> notes, as setting it higher could make a big difference in vacuum run >> times. > > +many. > > We're having this debate every release. I think the ongoing reticence to note > performance improvements in the release notes is hurting Postgres. > > For one, performance improvements are one of the prime reason users > upgrade. Without them being noted anywhere more dense than the commit log, > it's very hard to figure out what improved for users. A halfway widely > applicable performance improvement is far more impactful than many of the > feature changes we do list in the release notes. many++ > For another, it's also very frustrating for developers that focus on > performance. The reticence to note their work, while noting other, far > smaller, things in the release notes, pretty much tells us that our work isn't > valued. agreed -- Joe Conway PostgreSQL Contributors Team RDS Open Source Databases Amazon Web Services: https://aws.amazon.com
On Wed, May 15, 2024 at 11:48 PM Andres Freund <andres@anarazel.de> wrote: > > On 2024-05-15 10:38:20 +0200, Alvaro Herrera wrote: > > I disagree with this. IMO the impact of the Sawada/Naylor change is > > likely to be enormous for people with large tables and large numbers of > > tuples to clean up (I know we've had a number of customers in this > > situation, I can't imagine any Postgres service provider that doesn't). > > The fact that maintenance_work_mem is no longer capped at 1GB is very > > important and I think we should mention that explicitly in the release > > notes, as setting it higher could make a big difference in vacuum run > > times. > > +many. > > We're having this debate every release. I think the ongoing reticence to note > performance improvements in the release notes is hurting Postgres. > > For one, performance improvements are one of the prime reason users > upgrade. Without them being noted anywhere more dense than the commit log, > it's very hard to figure out what improved for users. A halfway widely > applicable performance improvement is far more impactful than many of the > feature changes we do list in the release notes. The practical reason this matters to users is that they want to change their configuration or expectations in response to performance improvements. And also, as Jelte mentions upthread, describing performance improvements made each release in Postgres makes it clear that we are consistently improving it. > For another, it's also very frustrating for developers that focus on > performance. The reticence to note their work, while noting other, far > smaller, things in the release notes, pretty much tells us that our work isn't > valued. +many - Melanie
On Wed, May 15, 2024 at 11:48 PM Andres Freund <andres@anarazel.de> wrote: > On 2024-05-15 10:38:20 +0200, Alvaro Herrera wrote: > > I disagree with this. IMO the impact of the Sawada/Naylor change is > > likely to be enormous for people with large tables and large numbers of > > tuples to clean up (I know we've had a number of customers in this > > situation, I can't imagine any Postgres service provider that doesn't). > > The fact that maintenance_work_mem is no longer capped at 1GB is very > > important and I think we should mention that explicitly in the release > > notes, as setting it higher could make a big difference in vacuum run > > times. > > +many. TIDStore/the lifting of the maintenance_work_mem cap is likely to make the performance of VACUUM a lot more predictable, overall. While most VACUUM operations don't hit the limit, the limit is disproportionately involved in cases where (for whatever reason) vacuuming becomes a long and painful process. Even if you as a user never run into such a problem, you still spend time worrying about it, and/or taking measures to make sure it doesn't affect you. The justification for not including mention of these items is that they're not very relevant to users. I find that hard to square with what does get included. For example, the "Source Code" section is full of highly niche items. Items that are low impact, even for users that'll benefit the most. Also, "Monitoring" often mentions monitoring improvements that expose low-level implementation details (e.g. SLRU statistics), even though there's a good chance that Bruce wouldn't include an item for some improvement to the SLRU subsystem itself. If somebody puts in an enormous amount of effort to get a big performance improvement over the line, then ISTM that that effort is a useful signal when the time comes to write the release notes (at least up to a point). For example, Masahiko and John spent about 2 years on the TIDStore thing, on and off. These things do not happen in a vacuum (no pun intended). Common sense tells me that they went to those lengths precisely because they understood that it very much was relevant to users. That belief would have been reinforced by both experience, and by discussion on the list during the development of the feature. To be fair to Bruce, it probably really is true that most individual users won't care about (say) TIDStore. But it's probably also true that most individual users don't care about the release notes, or at most skim the major items. -- Peter Geoghegan
On 2024-05-14 Tu 20:39, Bruce Momjian wrote: > On Sat, May 11, 2024 at 03:32:55PM -0400, Andrew Dunstan wrote: >> On 2024-05-09 Th 00:03, Bruce Momjian wrote: >>> I have committed the first draft of the PG 17 release notes; you can >>> see the results here: >>> >>> https://momjian.us/pgsql_docs/release-17.html >>> >>> It will be improved until the final release. The item count is 188, >>> which is similar to recent releases: >>> >>> release-10: 189 >>> release-11: 170 >>> release-12: 180 >>> release-13: 178 >>> release-14: 220 >>> release-15: 184 >>> release-16: 206 >>> release-17: 188 >>> >>> I welcome feedback. For some reason it was an easier job than usual. >> >> I don't like blowing my own horn but I feel commit 3311ea86ed "Introduce a >> non-recursive JSON parser" should be in the release notes. This isn't >> something that's purely internal, but it could be used by an extension or a >> client program to parse JSON documents that are too large to handle with the >> existing API. >> >> Maybe "Introduce an incremental JSON parser" would have been a better >> headline. > Well, this gets into a level of detail that is beyond the average > reader. I think at that level people will need to read the git logs or > review the code. Do we use it for anything yet? Yes, certainly, it's used in handling backup manifests. Without it we can't handle huge manifests. See commits ea7b4e9a2a and 222e11a10a. Other uses are in the works. cheers andrew -- Andrew Dunstan EDB: https://www.enterprisedb.com
On Thu, May 9, 2024 at 12:04 PM Bruce Momjian <bruce@momjian.us> wrote: > > I have committed the first draft of the PG 17 release notes; you can > see the results here: > > https://momjian.us/pgsql_docs/release-17.html > > It will be improved until the final release. The item count is 188, > which is similar to recent releases: > This thread mentioned performance. actually this[1] refactored some interval aggregation related functions, which will make these two aggregate function: avg(interval), sum(interval) run faster, especially avg(interval). see [2]. well, I guess, this is a kind of niche performance improvement to be mentioned separately. these 3 items need to be removed, because of https://git.postgresql.org/cgit/postgresql.git/commit/?id=8aee330af55d8a759b2b73f5a771d9d34a7b887f >> Add stratnum GiST support function (Paul A. Jungwirth) >> Allow foreign keys to reference WITHOUT OVERLAPS primary keys (Paul A. Jungwirth) >> The keyword PERIOD is used for this purpose. >> Allow PRIMARY KEY and UNIQUE constraints to use WITHOUT OVERLAPS for non-overlapping exclusion constraints (Paul A. Jungwirth) [1] https://git.postgresql.org/cgit/postgresql.git/commit/?id=519fc1bd9e9d7b408903e44f55f83f6db30742b7 [2] https://www.postgresql.org/message-id/CAEZATCUJ0xjyQUL7SHKxJ5a%2BDm5pjoq-WO3NtkDLi6c76rh58w%40mail.gmail.com
Bruce Momjian wrote: > I have committed the first draft of the PG 17 release notes; you can > see the results here: > > https://momjian.us/pgsql_docs/release-17.html About the changes in collations: <quote> Create a "builtin" collation provider similar to libc's C locale (Jeff Davis) It uses a "C" locale which is identical but independent of libc, but it allows the use of non-"C" collations like "en_US" and "C.UTF-8" with the "C" locale, which libc does not. MORE? </quote> The new builtin provider has two collations: * ucs_basic which is 100% identical to "C". It was introduced several versions ago and the v17 novelty is simply to change its pg_collation.collprovider from 'c' to 'b'. * pg_c_utf8 which sorts like "C" but is Unicode-aware for the rest, which makes it quite different from "C". It's also different from the other UTF-8 collations that could be used up to v17 in that it does not depend on an external library, making it free from the collation OS-upgrade risks. The part that is concretely of interest to users is the introduction of pg_c_utf8. As described in [1]: <quote> pg_c_utf8 This collation sorts by Unicode code point values rather than natural language order. For the functions lower, initcap, and upper, it uses Unicode simple case mapping. For pattern matching (including regular expressions), it uses the POSIX Compatible variant of Unicode Compatibility Properties. Behavior is efficient and stable within a Postgres major version. This collation is only available for encoding UTF8. </quote> I'd suggest that the relnote entry should be more like a condensed version of that description, without mentioning en_US or C.UTF-8, whose existence and semantics are OS-dependent, contrary to pg_c_utf8. [1] https://www.postgresql.org/docs/devel/collation.html Best regards, -- Daniel Vérité https://postgresql.verite.pro/ Twitter: @DanielVerite
On Thu, 2024-05-09 at 00:03 -0400, Bruce Momjian wrote: > I have committed the first draft of the PG 17 release notes; you can > see the results here: > > https://momjian.us/pgsql_docs/release-17.html For this item: Create a "builtin" collation provider similar to libc's C locale (Jeff Davis) It uses a "C" locale which is identical but independent of libc, but it allows the use of non-"C" collations like "en_US" and "C.UTF-8" with the "C" locale, which libc does not. MORE? I suggest something more like: New, platform-independent "builtin" collation provider. (Jeff Davis) Currently, it offers the "C" and "C.UTF-8" locales. The "C.UTF-8" locale combines stable and fast code point order collation with Unicode character semantics. Regards, Jeff Davis
On Thu, May 16, 2024 at 03:35:17PM +1200, David Rowley wrote: > On Thu, 16 May 2024 at 14:48, Bruce Momjian <bruce@momjian.us> wrote: > > > > On Wed, May 15, 2024 at 09:13:14AM -0400, Melanie Plageman wrote: > > > Also +1 on the Sawada/Naylor change being on the highlight section of > > > the release (as David suggested upthread). > > > > Agreed, I went with the attached applied patch. > > +Allow vacuum to more efficiently store tuple references and remove > its memory limit (Masahiko Sawada, John Naylor) > +</para> > > I don't want it to seem like I'm splitting hairs, but I'd drop the " > and remove its memory limit" > > +<para> > +Specifically, maintenance_work_mem and autovacuum_work_mem can now be > configured to use more than one gigabyte of memory. WAL traffic > caused by vacuum is also more compact. > > I'd say the first sentence above should be written as: > > "Additionally, vacuum no longer silently imposes a 1GB tuple reference > limit even when maintenance_work_mem or autovacuum_work_mem are set to > higher values" > > It's not "Specifically" as the "more efficiently store tuple > references" isn't the same thing as removing the 1GB cap. Also, there > was never a restriction in configuring maintenance_work_mem or > autovacuum_work_mem to values higher than 1GB. The restriction was > that vacuum was unable to utilize anything more than that. Slightly adjusted wording patch attached and applied. My deep apologies for the delay in addressing this. I should have done it sooner. -- Bruce Momjian <bruce@momjian.us> https://momjian.us EDB https://enterprisedb.com Only you can decide what is important to you.
Attachment
On Wed, May 15, 2024 at 08:48:02PM -0700, Andres Freund wrote: > Hi, > > On 2024-05-15 10:38:20 +0200, Alvaro Herrera wrote: > > I disagree with this. IMO the impact of the Sawada/Naylor change is > > likely to be enormous for people with large tables and large numbers of > > tuples to clean up (I know we've had a number of customers in this > > situation, I can't imagine any Postgres service provider that doesn't). > > The fact that maintenance_work_mem is no longer capped at 1GB is very > > important and I think we should mention that explicitly in the release > > notes, as setting it higher could make a big difference in vacuum run > > times. > > +many. > > We're having this debate every release. I think the ongoing reticence to note > performance improvements in the release notes is hurting Postgres. > > For one, performance improvements are one of the prime reason users > upgrade. Without them being noted anywhere more dense than the commit log, > it's very hard to figure out what improved for users. A halfway widely > applicable performance improvement is far more impactful than many of the > feature changes we do list in the release notes. I agree the impact of performance improvements are often greater than the average release note item. However, if people expect Postgres to be faster, is it important for them to know _why_ it is faster? If we add a new flag to a command, people will want to know about it so they can make use of it, or if there is a performance improvement that allows new workloads, they will want to know about that too so they can consider those workloads. On the flip side, a performance improvement that makes everything 10% faster has little behavioral change for users, and in fact I think we get ~6% faster in every major release. > For another, it's also very frustrating for developers that focus on > performance. The reticence to note their work, while noting other, far > smaller, things in the release notes, pretty much tells us that our work isn't > valued. Yes, but are we willing to add text that every user will have to read just for this purpose? One think we _could_ do is to create a generic performance release note item saying performance has been improved in the following areas, with no more details, but we can list the authors on the item. -- Bruce Momjian <bruce@momjian.us> https://momjian.us EDB https://enterprisedb.com Only you can decide what is important to you.
On Thu, May 16, 2024 at 09:09:11AM -0400, Melanie Plageman wrote: > On Wed, May 15, 2024 at 11:48 PM Andres Freund <andres@anarazel.de> wrote: > > > > On 2024-05-15 10:38:20 +0200, Alvaro Herrera wrote: > > > I disagree with this. IMO the impact of the Sawada/Naylor change is > > > likely to be enormous for people with large tables and large numbers of > > > tuples to clean up (I know we've had a number of customers in this > > > situation, I can't imagine any Postgres service provider that doesn't). > > > The fact that maintenance_work_mem is no longer capped at 1GB is very > > > important and I think we should mention that explicitly in the release > > > notes, as setting it higher could make a big difference in vacuum run > > > times. > > > > +many. > > > > We're having this debate every release. I think the ongoing reticence to note > > performance improvements in the release notes is hurting Postgres. > > > > For one, performance improvements are one of the prime reason users > > upgrade. Without them being noted anywhere more dense than the commit log, > > it's very hard to figure out what improved for users. A halfway widely > > applicable performance improvement is far more impactful than many of the > > feature changes we do list in the release notes. > > The practical reason this matters to users is that they want to change > their configuration or expectations in response to performance > improvements. > > And also, as Jelte mentions upthread, describing performance > improvements made each release in Postgres makes it clear that we are > consistently improving it. > > > For another, it's also very frustrating for developers that focus on > > performance. The reticence to note their work, while noting other, far > > smaller, things in the release notes, pretty much tells us that our work isn't > > valued. > > +many Please see the email I just posted. There are three goals we have to adjust for: 1. short release notes so they are readable 2. giving people credit for performance improvements 3. showing people Postgres cares about performance I would like to achieve 2 & 3 without harming #1. My experience is if I am reading a long document, and I get to a section where I start to wonder, "Why should I care about this?", I start to skim the rest of the document. I am particularly critical if I start to wonder, "Why does the author _think_ I should care about this?" becasue it feels like the author is writing for him/herself and not the audience. -- Bruce Momjian <bruce@momjian.us> https://momjian.us EDB https://enterprisedb.com Only you can decide what is important to you.
On Thu, May 16, 2024 at 04:29:38PM +0800, jian he wrote: > On Thu, May 9, 2024 at 12:04 PM Bruce Momjian <bruce@momjian.us> wrote: > > > > I have committed the first draft of the PG 17 release notes; you can > > see the results here: > > > > https://momjian.us/pgsql_docs/release-17.html > > > > >> Add jsonpath methods to convert JSON values to different data types (Jeevan Chalke) > >> The jsonpath methods are .bigint(), .boolean(), .date(), .decimal([precision [, scale]]), .integer(), .number(), .string(),.time(), .time_tz(), .timestamp(), and .timestamp_tz(). > > I think it's slightly incorrect. > > for example: > select jsonb_path_query('"2023-08-15"', '$.date()'); > I think it does is trying to validate json value "2023-08-15" can be a > date value, if so, print json string out, else error out. > > > "convert JSON values to different data types" > meaning that we are converting json values to another data type, date? I see your point. I have reworded it to be: Add jsonpath methods to convert JSON values to other JSON data types (Jeevan Chalke) Does that help? I think your example is causing confusion because once JSON values enter the SQL data type space, they are strings. -- Bruce Momjian <bruce@momjian.us> https://momjian.us EDB https://enterprisedb.com Only you can decide what is important to you.
On Thu, May 16, 2024 at 11:50:20AM -0400, Andrew Dunstan wrote: > > Maybe "Introduce an incremental JSON parser" would have been a better > > > headline. > > Well, this gets into a level of detail that is beyond the average > > reader. I think at that level people will need to read the git logs or > > review the code. Do we use it for anything yet? > > > Yes, certainly, it's used in handling backup manifests. Without it we can't > handle huge manifests. See commits ea7b4e9a2a and 222e11a10a. > > Other uses are in the works. Okay, added in the attached applied patch. -- Bruce Momjian <bruce@momjian.us> https://momjian.us EDB https://enterprisedb.com Only you can decide what is important to you.
Attachment
On 2024-05-18 Sa 12:50, Bruce Momjian wrote: > On Thu, May 16, 2024 at 11:50:20AM -0400, Andrew Dunstan wrote: >>> Maybe "Introduce an incremental JSON parser" would have been a better >>>> headline. >>> Well, this gets into a level of detail that is beyond the average >>> reader. I think at that level people will need to read the git logs or >>> review the code. Do we use it for anything yet? >> >> Yes, certainly, it's used in handling backup manifests. Without it we can't >> handle huge manifests. See commits ea7b4e9a2a and 222e11a10a. >> >> Other uses are in the works. > Okay, added in the attached applied patch. > Thanks cheers andrew -- Andrew Dunstan EDB: https://www.enterprisedb.com
On Fri, May 17, 2024 at 09:22:59PM +0800, jian he wrote: > On Thu, May 9, 2024 at 12:04 PM Bruce Momjian <bruce@momjian.us> wrote: > > > > I have committed the first draft of the PG 17 release notes; you can > > see the results here: > > > > https://momjian.us/pgsql_docs/release-17.html > > > > It will be improved until the final release. The item count is 188, > > which is similar to recent releases: > > > > This thread mentioned performance. > actually this[1] refactored some interval aggregation related functions, > which will make these two aggregate function: avg(interval), sum(interval) > run faster, especially avg(interval). > see [2]. > well, I guess, this is a kind of niche performance improvement to be > mentioned separately. > > > these 3 items need to be removed, because of > https://git.postgresql.org/cgit/postgresql.git/commit/?id=8aee330af55d8a759b2b73f5a771d9d34a7b887f > > >> Add stratnum GiST support function (Paul A. Jungwirth) > > >> Allow foreign keys to reference WITHOUT OVERLAPS primary keys (Paul A. Jungwirth) > >> The keyword PERIOD is used for this purpose. > > >> Allow PRIMARY KEY and UNIQUE constraints to use WITHOUT OVERLAPS for non-overlapping exclusion constraints (Paul A.Jungwirth) > > > [1] https://git.postgresql.org/cgit/postgresql.git/commit/?id=519fc1bd9e9d7b408903e44f55f83f6db30742b7 > [2] https://www.postgresql.org/message-id/CAEZATCUJ0xjyQUL7SHKxJ5a%2BDm5pjoq-WO3NtkDLi6c76rh58w%40mail.gmail.com Agreed, I have applied the attached patch to make the release notes current. -- Bruce Momjian <bruce@momjian.us> https://momjian.us EDB https://enterprisedb.com Only you can decide what is important to you.
Attachment
On Fri, May 17, 2024 at 01:30:03PM -0700, Jeff Davis wrote: > On Thu, 2024-05-09 at 00:03 -0400, Bruce Momjian wrote: > > I have committed the first draft of the PG 17 release notes; you can > > see the results here: > > > > https://momjian.us/pgsql_docs/release-17.html > > For this item: > > Create a "builtin" collation provider similar to libc's C > locale (Jeff Davis) > > It uses a "C" locale which is identical but independent of > libc, but it allows the use of non-"C" collations like "en_US" > and "C.UTF-8" with the "C" locale, which libc does not. MORE? > > I suggest something more like: > > New, platform-independent "builtin" collation > provider. (Jeff Davis) > > Currently, it offers the "C" and "C.UTF-8" locales. The > "C.UTF-8" locale combines stable and fast code point order > collation with Unicode character semantics. Okay, I went with the attached applied patch. Adjustments? -- Bruce Momjian <bruce@momjian.us> https://momjian.us EDB https://enterprisedb.com Only you can decide what is important to you.
Attachment
On Sun, 19 May 2024 at 02:40, Bruce Momjian <bruce@momjian.us> wrote: > > On Thu, May 16, 2024 at 03:35:17PM +1200, David Rowley wrote: > > "Additionally, vacuum no longer silently imposes a 1GB tuple reference > > limit even when maintenance_work_mem or autovacuum_work_mem are set to > > higher values" > Slightly adjusted wording patch attached and applied. Thanks for adjusting. It's a minor detail, but I'll mention it because you went to the effort to adjust it away from what I'd written... I didn't make a random choice to use "or" between the two GUCs. Changing it to "and", IMO, isn't an improvement. Using "and" implies that the silent limited was only imposed when both of these GUCs were set >= 1GB. That's not true. For the case we're talking about here, if autovacuum_work_mem is set to anything apart from -1 then the value of maintenance_work_mem does not matter. David
On Sun, May 19, 2024 at 03:53:38PM +1200, David Rowley wrote: > On Sun, 19 May 2024 at 02:40, Bruce Momjian <bruce@momjian.us> wrote: > > > > On Thu, May 16, 2024 at 03:35:17PM +1200, David Rowley wrote: > > > "Additionally, vacuum no longer silently imposes a 1GB tuple reference > > > limit even when maintenance_work_mem or autovacuum_work_mem are set to > > > higher values" > > > Slightly adjusted wording patch attached and applied. > > Thanks for adjusting. > > It's a minor detail, but I'll mention it because you went to the > effort to adjust it away from what I'd written... True. > I didn't make a random choice to use "or" between the two GUCs. > Changing it to "and", IMO, isn't an improvement. Using "and" implies > that the silent limited was only imposed when both of these GUCs were > set >= 1GB. That's not true. For the case we're talking about here, if > autovacuum_work_mem is set to anything apart from -1 then the value of > maintenance_work_mem does not matter. Okay, changed to "or". -- Bruce Momjian <bruce@momjian.us> https://momjian.us EDB https://enterprisedb.com Only you can decide what is important to you.
Hi Bruce, thanks for doing this again! I'm a bit late to this discussion -- there's been a bit of churn in the vacuum items, and some streams got crossed along the way. I've attached an attempt to rectify this.
Attachment
On Mon, May 20, 2024 at 01:23:02PM +0700, John Naylor wrote: > Hi Bruce, thanks for doing this again! > > I'm a bit late to this discussion -- there's been a bit of churn in > the vacuum items, and some streams got crossed along the way. I've > attached an attempt to rectify this. Agreed, patch applied, thanks. -- Bruce Momjian <bruce@momjian.us> https://momjian.us EDB https://enterprisedb.com Only you can decide what is important to you.
On Sat, May 18, 2024 at 11:13 AM Bruce Momjian <bruce@momjian.us> wrote: > > On Thu, May 16, 2024 at 09:09:11AM -0400, Melanie Plageman wrote: > > On Wed, May 15, 2024 at 11:48 PM Andres Freund <andres@anarazel.de> wrote: > > > > > > On 2024-05-15 10:38:20 +0200, Alvaro Herrera wrote: > > > > I disagree with this. IMO the impact of the Sawada/Naylor change is > > > > likely to be enormous for people with large tables and large numbers of > > > > tuples to clean up (I know we've had a number of customers in this > > > > situation, I can't imagine any Postgres service provider that doesn't). > > > > The fact that maintenance_work_mem is no longer capped at 1GB is very > > > > important and I think we should mention that explicitly in the release > > > > notes, as setting it higher could make a big difference in vacuum run > > > > times. > > > > > > +many. > > > > > > We're having this debate every release. I think the ongoing reticence to note > > > performance improvements in the release notes is hurting Postgres. > > > > > > For one, performance improvements are one of the prime reason users > > > upgrade. Without them being noted anywhere more dense than the commit log, > > > it's very hard to figure out what improved for users. A halfway widely > > > applicable performance improvement is far more impactful than many of the > > > feature changes we do list in the release notes. > > > > The practical reason this matters to users is that they want to change > > their configuration or expectations in response to performance > > improvements. > > > > And also, as Jelte mentions upthread, describing performance > > improvements made each release in Postgres makes it clear that we are > > consistently improving it. > > > > > For another, it's also very frustrating for developers that focus on > > > performance. The reticence to note their work, while noting other, far > > > smaller, things in the release notes, pretty much tells us that our work isn't > > > valued. > > > > +many > > Please see the email I just posted. There are three goals we have to > adjust for: > > 1. short release notes so they are readable > 2. giving people credit for performance improvements > 3. showing people Postgres cares about performance I agree with all three of these goals. I would even add to 3 "show users Postgres is addressing their performance complaints". That in particular makes me less excited about having a "generic performance release note item saying performance has been improved in the following areas" (from your other email). I think that describing the specific performance improvements is required to 1) allow users to change expectations and configurations to take advantage of the performance enhancements 2) ensure users know that their performance concerns are being addressed. > I would like to achieve 2 & 3 without harming #1. My experience is if I > am reading a long document, and I get to a section where I start to > wonder, "Why should I care about this?", I start to skim the rest of > the document. I am particularly critical if I start to wonder, "Why > does the author _think_ I should care about this?" becasue it feels like > the author is writing for him/herself and not the audience. I see what you are saying. We don't want to just end up with the whole git log in the release notes. However, we know that not all users will care about the same features. As someone said somewhere else in this thread, presumably hackers spend time on features because some users want them. - Melanie
On Mon, May 20, 2024 at 02:35:37PM -0400, Melanie Plageman wrote: > On Sat, May 18, 2024 at 11:13 AM Bruce Momjian <bruce@momjian.us> wrote: > > Please see the email I just posted. There are three goals we have to > > adjust for: > > > > 1. short release notes so they are readable > > 2. giving people credit for performance improvements > > 3. showing people Postgres cares about performance > > I agree with all three of these goals. I would even add to 3 "show > users Postgres is addressing their performance complaints". That in > particular makes me less excited about having a "generic performance > release note item saying performance has been improved in the > following areas" (from your other email). I think that describing the > specific performance improvements is required to 1) allow users to > change expectations and configurations to take advantage of the > performance enhancements 2) ensure users know that their performance > concerns are being addressed. Well, as you can see, doing #2 & #3 works against accomplishing #1. > > I would like to achieve 2 & 3 without harming #1. My experience is if I > > am reading a long document, and I get to a section where I start to > > wonder, "Why should I care about this?", I start to skim the rest of > > the document. I am particularly critical if I start to wonder, "Why > > does the author _think_ I should care about this?" becasue it feels like > > the author is writing for him/herself and not the audience. > > I see what you are saying. We don't want to just end up with the whole > git log in the release notes. However, we know that not all users will > care about the same features. As someone said somewhere else in this > thread, presumably hackers spend time on features because some users > want them. I think we need as a separate section about performance improvements that don't affect specific workloads. Peter Eisentraut created an Acknowledgements section at the bottom of the release notes, similar to #2 above, so maybe someone else can add a performance internals section too. -- Bruce Momjian <bruce@momjian.us> https://momjian.us EDB https://enterprisedb.com Only you can decide what is important to you.
On Mon, May 20, 2024 at 9:37 AM Bruce Momjian <bruce@momjian.us> wrote: > > On Mon, May 20, 2024 at 01:23:02PM +0700, John Naylor wrote: > > Hi Bruce, thanks for doing this again! > > > > I'm a bit late to this discussion -- there's been a bit of churn in > > the vacuum items, and some streams got crossed along the way. I've > > attached an attempt to rectify this. > > Agreed, patch applied, thanks. If "Allow vacuum to more efficiently remove and freeze tuples" stays in the release notes, I would add Heikki's name. He wasn't listed as a co-author on all of the commits that were part of this feature, but he made a substantial investment in it and should be listed, IMO. Patch attached. - Melanie
Attachment
On Sat, 2024-05-18 at 17:51 -0400, Bruce Momjian wrote:
> Okay, I went with the attached applied patch. Adjustments?
I think it should have more emphasis on the actual new feature: a
platform-independent builtin collation provider and the C.UTF-8 locale.
The user can look at the documentation for comparison with libc.
Regards,
Jeff Davis
On Thu, May 9, 2024 at 1:04 PM Bruce Momjian <bruce@momjian.us> wrote: > I have committed the first draft of the PG 17 release notes; you can > see the results here: > > https://momjian.us/pgsql_docs/release-17.html > > It will be improved until the final release. The item count is 188, > which is similar to recent releases: > > release-10: 189 > release-11: 170 > release-12: 180 > release-13: 178 > release-14: 220 > release-15: 184 > release-16: 206 > release-17: 188 > > I welcome feedback. For some reason it was an easier job than usual. Thanks Bruce for working on this as always. Failed to notice when I read the notes before: <listitem> <para> Add SQL/JSON constructor functions JSON(), JSON_SCALAR(), and JSON_SERIALIZE() (Amit Langote) </para> </listitem> Should be: <listitem> <para> Add SQL/JSON constructor functions JSON(), JSON_SCALAR(), and JSON_SERIALIZE() (Nikita Glukhov, Teodor Sigaev, Oleg Bartunov, Alexander Korotkov, Andrew Dunstan, Amit Langote) </para> </listitem> Patch attached. -- Thanks, Amit Langote
Attachment
Hi, On 2024-05-18 10:59:47 -0400, Bruce Momjian wrote: > On Wed, May 15, 2024 at 08:48:02PM -0700, Andres Freund wrote: > > +many. > > > > We're having this debate every release. I think the ongoing reticence to note > > performance improvements in the release notes is hurting Postgres. > > > > For one, performance improvements are one of the prime reason users > > upgrade. Without them being noted anywhere more dense than the commit log, > > it's very hard to figure out what improved for users. A halfway widely > > applicable performance improvement is far more impactful than many of the > > feature changes we do list in the release notes. > > I agree the impact of performance improvements are often greater than > the average release note item. However, if people expect Postgres to be > faster, is it important for them to know _why_ it is faster? Yes, it very often is. Performance improvements typically aren't "make everything 3% faster", they're more "make this special thing 20% faster". Without know what got faster, users don't know if a) the upgrade will improve their production situation b) they need to change something to take advantage of the improvement > On the flip side, a performance improvement that makes everything 10% > faster has little behavioral change for users, and in fact I think we > get ~6% faster in every major release. I cannot recall many "make everything faster" improvements, if any. And even if it's "make everything faster" - that's useful for users to know, it might solve their production problem! It's also good for PR. Given how expensive postgres upgrades still are, we can't expect production workloads to upgrade to every major version. The set of performance improvements and feature additions between major versions can help users make an informed decision. Also, the release notes are also not just important to users. I often go back and look in the release notes to see when some some important change was made, because sometimes it's harder to find it in the git log, due to sheer volume. And even just keeping up with all the changes between two releases is hard, it's useful to be able to read the release notes and see what happened. > > For another, it's also very frustrating for developers that focus on > > performance. The reticence to note their work, while noting other, far > > smaller, things in the release notes, pretty much tells us that our work isn't > > valued. > > Yes, but are we willing to add text that every user will have to read > just for this purpose? Of course it's a tradeoff. We shouldn't verbosely note down every small changes just because of the recognition, that'd make the release notes unreadable. And it'd just duplicate the commit log. But that's not the same as defaulting to not noting performance improvements, even if the performance improvement is more impactful than many other features that are noted. > One think we _could_ do is to create a generic performance release note > item saying performance has been improved in the following areas, with > no more details, but we can list the authors on the item. To me that's the "General Performance" section. If somebody reading the release notes doesn't care about performance, they can just skip that section ([1]). I don't see why we wouldn't want to include the same level of detail as for other changes. Greetings, Andres Freund [1] I've wondered if we should have one more level of TOC on the release note page, so it's easier to jump to specific sections.
Hi,
On 2024-05-18 11:13:54 -0400, Bruce Momjian wrote:
> Please see the email I just posted. There are three goals we have to
> adjust for:
>
> 1. short release notes so they are readable
> 2. giving people credit for performance improvements
> 3. showing people Postgres cares about performance
>
> I would like to achieve 2 & 3 without harming #1. My experience is if I
> am reading a long document, and I get to a section where I start to
> wonder, "Why should I care about this?", I start to skim the rest of
> the document.
I agree keeping things reasonably short is important. But I don't think you're
evenly applying it as a goal.
Just skimming the notes from the end, I see
- an 8 entries long pg_stat_statements section
- multiple entries about "Create custom wait events for ..."
- three entries about adding --all to {reindexdb,vacuumdb,clusterdb}.
- an entry about adding long options to pg_archivecleanup
- two entries about grantable maintenance rights, once via pg_maintain, once
per-table
- separate entries about pg_stat_reset_slru(), pg_stat_reset_shared("slru"),
If you're concerned about brevity, we can make things shorter without skipping
over most performance imporvements.
> I am particularly critical if I start to wonder, "Why
> does the author _think_ I should care about this?" becasue it feels like
> the author is writing for him/herself and not the audience.
FWIW, I think it's a good thing for somebody other than the author to have a
hand in writing a release notes entry for a change. The primary author(s) are
often too deep into some issue to have a good view of the right level of
detail and understandability.
Greetings,
Andres Freund
Hi, On 2024-05-21 09:27:20 -0700, Andres Freund wrote: > Also, the release notes are also not just important to users. I often go back > and look in the release notes to see when some some important change was made, > because sometimes it's harder to find it in the git log, due to sheer > volume. And even just keeping up with all the changes between two releases is > hard, it's useful to be able to read the release notes and see what happened. > > [...] > > [1] I've wondered if we should have one more level of TOC on the release note > page, so it's easier to jump to specific sections. Which reminds me: Eventually I'd like to add links to the most important commits related to release note entries. We already do much of the work of building that list of commits for each entry. That'd allow a reader to find more details if interested. Right one either has to open the sgml file (which no user will know to do), or find the git entries manually. The latter of which is often hard, because the git commits often will use different wording etc. Admittedly doing so within the constraints of docbook and not wanting to overly decrease density (both in .sgml and the resulting html) isn't a trivial task. That's entirely independent of my concern around noting performance improvements in the release notes, of course. Greetings, Andres Freund
On 2024-May-21, Andres Freund wrote: > Which reminds me: Eventually I'd like to add links to the most important > commits related to release note entries. We already do much of the work of > building that list of commits for each entry. That'd allow a reader to find > more details if interested. +1. Several times I have had to open the SGML to fetch a commit ID and build a URL to provide to someone. -- Álvaro Herrera Breisgau, Deutschland — https://www.EnterpriseDB.com/ "Puedes vivir sólo una vez, pero si lo haces bien, una vez es suficiente"
On Tue, May 21, 2024 at 12:27 PM Andres Freund <andres@anarazel.de> wrote: > > I agree the impact of performance improvements are often greater than > > the average release note item. However, if people expect Postgres to be > > faster, is it important for them to know _why_ it is faster? > > Yes, it very often is. Is it important for them to even read the release notes? Bruce's arguments against listing performance items more often/with greater prominence could just as easily be applied to other types of features, in other areas. Performance is a feature (or a feature category) -- no better or worse than any other category of feature. > Performance improvements typically aren't "make > everything 3% faster", they're more "make this special thing 20% > faster". Without know what got faster, users don't know if > a) the upgrade will improve their production situation > b) they need to change something to take advantage of the improvement Another important category of performance improvement is "make the thing that was just unusable usable, for the first time ever". Sometimes the baseline is unreasonably slow, so an improvement effectively allows you as a user to do something that just wasn't possible on previous versions. Other times it's addressed at something that was very scary, like VACUUMs that need multiple rounds of index vacuuming. Multiple rounds of index vacuuming are just woefully, horribly inefficient, and are the single individual thing that can make things far worse. Even if you didn't technically have that problem before now, you did have the problem of having to worry about it. So the work in question has sanded-down this really nasty sharp edge. That's a substantial quality of life improvement for many users. In short, many individual performance improvements are best thought of as qualitative improvements, rather than quantitative improvements. It doesn't help that there is a kind of pressure to present them as quantitative improvements. For example, I was recently encouraged to present my own Postgres 17 B-Tree work internally using some kind of headline grabbing measure like "6x faster". That just seems silly to me. I can contrive a case where it's faster by an arbitrarily large amount. Much like how a selective index scan can be arbitrarily faster than a sequential scan. Again, a qualitative improvement. -- Peter Geoghegan
On Tue, May 21, 2024 at 12:27 PM Andres Freund <andres@anarazel.de> wrote:
> To me that's the "General Performance" section. If somebody reading the
> release notes doesn't care about performance, they can just skip that section
> ([1]). I don't see why we wouldn't want to include the same level of detail
> as for other changes.
I'm relatively sure that we've had this argument in previous years and
essentially everyone but Bruce has agreed with the idea that
performance changes ought to be treated the same as any other kind of
improvement. The difficulty is that Bruce is the one doing the release
notes. I think it might help if someone were willing to prepare a
patch showing what they think specifically should be changed. Or maybe
Bruce would be willing to provide a list of all of the performance
improvements he doesn't think are worth release-noting or isn't sure
how to release-note, and someone else can then have a go at them.
Personally, I suspect that a part of the problem, other than the
inevitable fact that the person doing the work has a perspective on
how the work should be done with which not everyone will agree, is
that a lot of performance changes have commit messages that don't
really explain what the user impact is. For instance, consider
6dbb490261a6170a3fc3e326c6983ad63e795047 ("Combine freezing and
pruning steps in VACUUM"). It does actually say what the benefit is
("That reduces the overall amount of WAL generated") but the reader
could easily be left wondering whether that is really the selling
point. Does it also reduce CPU consumption? Is that more or less
important than the WAL reduction? Was the WAL reduction the motivation
for the work? Is the WAL reduction significant enough that this is a
feature in its own right, or is this just preparatory to some other
work? These kinds of ambiguities can exist for any commit, not just
performance commits, but I bet that on average the problem is worse
for performance-related commits.
--
Robert Haas
EDB: http://www.enterprisedb.com
On Tue, May 21, 2024 at 1:51 PM Robert Haas <robertmhaas@gmail.com> wrote:
>
> On Tue, May 21, 2024 at 12:27 PM Andres Freund <andres@anarazel.de> wrote:
> > To me that's the "General Performance" section. If somebody reading the
> > release notes doesn't care about performance, they can just skip that section
> > ([1]). I don't see why we wouldn't want to include the same level of detail
> > as for other changes.
>
> I'm relatively sure that we've had this argument in previous years and
> essentially everyone but Bruce has agreed with the idea that
> performance changes ought to be treated the same as any other kind of
> improvement. The difficulty is that Bruce is the one doing the release
> notes. I think it might help if someone were willing to prepare a
> patch showing what they think specifically should be changed. Or maybe
> Bruce would be willing to provide a list of all of the performance
> improvements he doesn't think are worth release-noting or isn't sure
> how to release-note, and someone else can then have a go at them.
>
> Personally, I suspect that a part of the problem, other than the
> inevitable fact that the person doing the work has a perspective on
> how the work should be done with which not everyone will agree, is
> that a lot of performance changes have commit messages that don't
> really explain what the user impact is. For instance, consider
> 6dbb490261a6170a3fc3e326c6983ad63e795047 ("Combine freezing and
> pruning steps in VACUUM"). It does actually say what the benefit is
> ("That reduces the overall amount of WAL generated") but the reader
> could easily be left wondering whether that is really the selling
> point. Does it also reduce CPU consumption? Is that more or less
> important than the WAL reduction? Was the WAL reduction the motivation
> for the work? Is the WAL reduction significant enough that this is a
> feature in its own right, or is this just preparatory to some other
> work? These kinds of ambiguities can exist for any commit, not just
> performance commits, but I bet that on average the problem is worse
> for performance-related commits.
In Postgres development, we break larger projects into smaller ones
and then those smaller projects into multiple individual commits. Each
commit needs to stand alone and each subproject needs to have a
defensible benefit. One thing that is harder with performance-related
work than non-performance feature work is that there isn't always a
final "turn it on" commit. For example, let's say you are adding a new
view that tracks new stats of some kind. You do a bunch of refactoring
and small subprojects to make it possible to add the view. Then the
final commit that actually creates the view has obvious user value to
whoever is reading the log. For performance features, it doesn't
always work like this.
For the vacuum WAL volume reduction, there were a bunch of smaller
projects throughout the last development year that I worked on that
were committed by different people and with different individual
benefits. Some changes caused vacuum to do less visibility checks (so
less CPU usage), some changed WAL format in a way that saves some
space, and some, like the commit you mention, make vacuum emit less
WAL. That commit by itself doesn't contain all of the user benefits of
the whole project. I couldn't think of a good place to list all of the
commits together that were part of the same project. Perhaps you could
argue that they were not in fact part of the same project and instead
were just small individual changes -- none of which are individually
worth including in the release notes.
- Melanie
On Tue, May 21, 2024 at 2:26 PM Melanie Plageman <melanieplageman@gmail.com> wrote: > For the vacuum WAL volume reduction, there were a bunch of smaller > projects throughout the last development year that I worked on that > were committed by different people and with different individual > benefits. Some changes caused vacuum to do less visibility checks (so > less CPU usage), some changed WAL format in a way that saves some > space, and some, like the commit you mention, make vacuum emit less > WAL. That commit by itself doesn't contain all of the user benefits of > the whole project. I couldn't think of a good place to list all of the > commits together that were part of the same project. Perhaps you could > argue that they were not in fact part of the same project and instead > were just small individual changes -- none of which are individually > worth including in the release notes. Yeah, I think a lot of projects have this problem in one way or another, but I think it may be worse for performance-related projects. I wasn't intending to knock that particular commit, just to be clear, or the commit message. I'm just saying that sometimes summarizing the commit log may not be as easy as we'd hope. -- Robert Haas EDB: http://www.enterprisedb.com
Hi, On Thu, May 9, 2024 at 1:03 PM Bruce Momjian <bruce@momjian.us> wrote: > > I have committed the first draft of the PG 17 release notes; you can > see the results here: > > https://momjian.us/pgsql_docs/release-17.html > > It will be improved until the final release. The item count is 188, > which is similar to recent releases: > I found a typo: s/pg_statstatement/pg_stat_statement/ I've attached a patch to fix it. Regards, -- Masahiko Sawada Amazon Web Services: https://aws.amazon.com
Attachment
On 2024-05-09 13:03, Bruce Momjian wrote: > I have committed the first draft of the PG 17 release notes; you can > see the results here: > > https://momjian.us/pgsql_docs/release-17.html > > It will be improved until the final release. The item count is 188, > which is similar to recent releases: > > release-10: 189 > release-11: 170 > release-12: 180 > release-13: 178 > release-14: 220 > release-15: 184 > release-16: 206 > release-17: 188 > > I welcome feedback. For some reason it was an easier job than usual. Thanks for working on this as always. <para> Allow pg_stat_reset_shared("slru") to clear SLRU statistics (Atsushi Torikoshi) </para> Considering someone may copy and paste this, 'slru' is better than "slru", isn't it? I also found an older release note[1] used single quotes for this like: Add pg_stat_reset_shared('bgwriter') to reset the cluster-wide shared statistics for the background writer (Greg Smith) [1] https://www.postgresql.org/docs/release/9.0.0/ -- Regards, -- Atsushi Torikoshi NTT DATA Group Corporation
Attachment
On Mon, May 20, 2024 at 02:47:28PM -0400, Melanie Plageman wrote: > On Mon, May 20, 2024 at 9:37 AM Bruce Momjian <bruce@momjian.us> wrote: > > > > On Mon, May 20, 2024 at 01:23:02PM +0700, John Naylor wrote: > > > Hi Bruce, thanks for doing this again! > > > > > > I'm a bit late to this discussion -- there's been a bit of churn in > > > the vacuum items, and some streams got crossed along the way. I've > > > attached an attempt to rectify this. > > > > Agreed, patch applied, thanks. > > If "Allow vacuum to more efficiently remove and freeze tuples" stays > in the release notes, I would add Heikki's name. He wasn't listed as a > co-author on all of the commits that were part of this feature, but he > made a substantial investment in it and should be listed, IMO. Patch > attached. Thanks, patch applied. -- Bruce Momjian <bruce@momjian.us> https://momjian.us EDB https://enterprisedb.com Only you can decide what is important to you.
On Tue, May 21, 2024 at 09:27:20AM -0700, Andres Freund wrote: > On 2024-05-18 10:59:47 -0400, Bruce Momjian wrote: > > I agree the impact of performance improvements are often greater than > > the average release note item. However, if people expect Postgres to be > > faster, is it important for them to know _why_ it is faster? > > Yes, it very often is. Performance improvements typically aren't "make > everything 3% faster", they're more "make this special thing 20% > faster". Without know what got faster, users don't know if > a) the upgrade will improve their production situation > b) they need to change something to take advantage of the improvement You might have seen in this thread, I do record commits that speed up workloads that are user-visible, or specifically make new workloads possible. I assume that covers the items above, though I have to determine this from the commit message. > > On the flip side, a performance improvement that makes everything 10% > > faster has little behavioral change for users, and in fact I think we > > get ~6% faster in every major release. > > I cannot recall many "make everything faster" improvements, if any. > > And even if it's "make everything faster" - that's useful for users to know, > it might solve their production problem! It's also good for PR. Again, it is down to having three goals for the release notes, and #1 is having it readable/short, and 2 & 3 are for PR and acknowledging authors. > Also, the release notes are also not just important to users. I often go back > and look in the release notes to see when some some important change was made, > because sometimes it's harder to find it in the git log, due to sheer > volume. And even just keeping up with all the changes between two releases is > hard, it's useful to be able to read the release notes and see what happened. Well, I would say we need some _other_ way to record and perhaps advertise such changes. > > > For another, it's also very frustrating for developers that focus on > > > performance. The reticence to note their work, while noting other, far > > > smaller, things in the release notes, pretty much tells us that our work isn't > > > valued. > > > > Yes, but are we willing to add text that every user will have to read > > just for this purpose? > > Of course it's a tradeoff. We shouldn't verbosely note down every small > changes just because of the recognition, that'd make the release notes > unreadable. And it'd just duplicate the commit log. But that's not the same as > defaulting to not noting performance improvements, even if the performance > improvement is more impactful than many other features that are noted. Again, see above, I do mention performance improvements, but they have to be user-visible or enable new workloads. -- Bruce Momjian <bruce@momjian.us> https://momjian.us EDB https://enterprisedb.com Only you can decide what is important to you.
On Tue, May 21, 2024 at 09:51:09AM -0700, Andres Freund wrote: > Hi, > > On 2024-05-21 09:27:20 -0700, Andres Freund wrote: > > Also, the release notes are also not just important to users. I often go back > > and look in the release notes to see when some some important change was made, > > because sometimes it's harder to find it in the git log, due to sheer > > volume. And even just keeping up with all the changes between two releases is > > hard, it's useful to be able to read the release notes and see what happened. > > > > [...] > > > > [1] I've wondered if we should have one more level of TOC on the release note > > page, so it's easier to jump to specific sections. > > Which reminds me: Eventually I'd like to add links to the most important > commits related to release note entries. We already do much of the work of > building that list of commits for each entry. That'd allow a reader to find > more details if interested. Yes, it would be cool if they could mouse-over a graphic next to each release note item to get a popup to the commits. -- Bruce Momjian <bruce@momjian.us> https://momjian.us EDB https://enterprisedb.com Only you can decide what is important to you.
On Tue, May 21, 2024 at 01:50:58PM -0400, Robert Haas wrote:
> On Tue, May 21, 2024 at 12:27 PM Andres Freund <andres@anarazel.de> wrote:
> > To me that's the "General Performance" section. If somebody reading the
> > release notes doesn't care about performance, they can just skip that section
> > ([1]). I don't see why we wouldn't want to include the same level of detail
> > as for other changes.
>
> I'm relatively sure that we've had this argument in previous years and
> essentially everyone but Bruce has agreed with the idea that
> performance changes ought to be treated the same as any other kind of
> improvement. The difficulty is that Bruce is the one doing the release
> notes. I think it might help if someone were willing to prepare a
> patch showing what they think specifically should be changed. Or maybe
> Bruce would be willing to provide a list of all of the performance
> improvements he doesn't think are worth release-noting or isn't sure
> how to release-note, and someone else can then have a go at them.
Well, developers do ask why their individual commits were not listed,
and I repeat the same thing, as I have done in this thread multiple
times. You can probably look over this thread to find them, or in
previous years.
To be clear, it is performance improvements that don't have user-visible
changes or enable new workloads that I skip listing. I will also note
that don't remember any user ever finding a performance boost, and then
coming to use and asking why we didn't list it --- this release note
review process seems to add all of those.
Maybe adding a section called "Internal Performance". Here is our
General Performance contents:
E.1.3.1.3. General Performance
Allow vacuum to more efficiently remove and freeze tuples (Melanie
Plageman)
WAL traffic caused by vacuum is also more compact.
Allow vacuum to more efficiently store tuple references (Masahiko
Sawada, John Naylor)
Additionally, vacuum is no longer silently limited to one gigabyte
of memory when maintenance_work_mem or autovacuum_work_mem are higher.
Optimize vacuuming of relations with no indexes (Melanie Plageman)
Increase default vacuum_buffer_usage_limit to 2MB (Thomas Munro)
Improve performance when checking roles with many memberships
(Nathan Bossart)
Improve performance of heavily-contended WAL writes (Bharath
Rupireddy)
Improve performance when transferring large blocks of data to a
client (Melih Mutlu)
Allow the grouping of file system reads with the new system variable
io_combine_limit (Thomas Munro, Andres Freund, Melanie Plageman, Nazir
Bilal Yavuz)
Do we think more user-invisible changes should be in there?
--
Bruce Momjian <bruce@momjian.us> https://momjian.us
EDB https://enterprisedb.com
Only you can decide what is important to you.
On Tue, May 21, 2024 at 02:26:15PM -0400, Melanie Plageman wrote: > In Postgres development, we break larger projects into smaller ones > and then those smaller projects into multiple individual commits. Each > commit needs to stand alone and each subproject needs to have a > defensible benefit. One thing that is harder with performance-related > work than non-performance feature work is that there isn't always a > final "turn it on" commit. For example, let's say you are adding a new > view that tracks new stats of some kind. You do a bunch of refactoring > and small subprojects to make it possible to add the view. Then the > final commit that actually creates the view has obvious user value to > whoever is reading the log. For performance features, it doesn't > always work like this. > > For the vacuum WAL volume reduction, there were a bunch of smaller > projects throughout the last development year that I worked on that > were committed by different people and with different individual > benefits. Some changes caused vacuum to do less visibility checks (so > less CPU usage), some changed WAL format in a way that saves some > space, and some, like the commit you mention, make vacuum emit less > WAL. That commit by itself doesn't contain all of the user benefits of > the whole project. I couldn't think of a good place to list all of the > commits together that were part of the same project. Perhaps you could > argue that they were not in fact part of the same project and instead > were just small individual changes -- none of which are individually > worth including in the release notes. I try and group them, but I am sure imperfectly. It is very true that infrastucture changes that enable later commits are often missed. -- Bruce Momjian <bruce@momjian.us> https://momjian.us EDB https://enterprisedb.com Only you can decide what is important to you.
On Tue, May 21, 2024 at 09:40:28AM -0700, Andres Freund wrote:
> Hi,
>
> On 2024-05-18 11:13:54 -0400, Bruce Momjian wrote:
> > Please see the email I just posted. There are three goals we have to
> > adjust for:
> >
> > 1. short release notes so they are readable
> > 2. giving people credit for performance improvements
> > 3. showing people Postgres cares about performance
> >
> > I would like to achieve 2 & 3 without harming #1. My experience is if I
> > am reading a long document, and I get to a section where I start to
> > wonder, "Why should I care about this?", I start to skim the rest of
> > the document.
>
> I agree keeping things reasonably short is important. But I don't think you're
> evenly applying it as a goal.
>
> Just skimming the notes from the end, I see
> - an 8 entries long pg_stat_statements section
What item did you want to remove? Those are all user-visible changes.
> - multiple entries about "Create custom wait events for ..."
Well, those are all in different sections, so how can they be merged,
unless I create a "wait event section", I guess.
> - three entries about adding --all to {reindexdb,vacuumdb,clusterdb}.
The problem with merging these is that the "Specifically, --all can now
be used with" is different for all three of them.
> - an entry about adding long options to pg_archivecleanup
Well, that is a user-visible change. Should it not be listed?
> - two entries about grantable maintenance rights, once via pg_maintain, once
> per-table
Well, one is a GRANT and another is a role, so merging them seemed like
it would be too confusing.
> - separate entries about pg_stat_reset_slru(), pg_stat_reset_shared("slru"),
They are different functions with different detail text.
--
Bruce Momjian <bruce@momjian.us> https://momjian.us
EDB https://enterprisedb.com
Only you can decide what is important to you.
On Mon, May 20, 2024 at 11:48:09AM -0700, Jeff Davis wrote: > On Sat, 2024-05-18 at 17:51 -0400, Bruce Momjian wrote: > > Okay, I went with the attached applied patch. Adjustments? > > I think it should have more emphasis on the actual new feature: a > platform-independent builtin collation provider and the C.UTF-8 locale. > > The user can look at the documentation for comparison with libc. Okay, changed with the attached applied patch. -- Bruce Momjian <bruce@momjian.us> https://momjian.us EDB https://enterprisedb.com Only you can decide what is important to you.
Attachment
On Tue, May 21, 2024 at 11:20:02AM +0900, Amit Langote wrote: > Thanks Bruce for working on this as always. > > Failed to notice when I read the notes before: > > <listitem> > <para> > Add SQL/JSON constructor functions JSON(), JSON_SCALAR(), and > JSON_SERIALIZE() (Amit Langote) > </para> > </listitem> > > Should be: > > <listitem> > <para> > Add SQL/JSON constructor functions JSON(), JSON_SCALAR(), and > JSON_SERIALIZE() (Nikita Glukhov, Teodor Sigaev, Oleg Bartunov, > Alexander Korotkov, Andrew Dunstan, Amit Langote) > </para> > </listitem> Thanks, applied. -- Bruce Momjian <bruce@momjian.us> https://momjian.us EDB https://enterprisedb.com Only you can decide what is important to you.
On Wed, May 22, 2024 at 11:29:06AM +0900, Masahiko Sawada wrote: > I found a typo: > > s/pg_statstatement/pg_stat_statement/ > > I've attached a patch to fix it. Agreed, applied, thanks. -- Bruce Momjian <bruce@momjian.us> https://momjian.us EDB https://enterprisedb.com Only you can decide what is important to you.
On Wed, May 22, 2024 at 09:25:41PM +0900, torikoshia wrote:
> Thanks for working on this as always.
>
> <para>
> Allow pg_stat_reset_shared("slru") to clear SLRU statistics (Atsushi
> Torikoshi)
> </para>
>
> Considering someone may copy and paste this, 'slru' is better than "slru",
> isn't it?
> I also found an older release note[1] used single quotes for this like:
>
> Add pg_stat_reset_shared('bgwriter') to reset the cluster-wide shared
> statistics for the background writer (Greg Smith)
Agreed, patch applied, thanks.
--
Bruce Momjian <bruce@momjian.us> https://momjian.us
EDB https://enterprisedb.com
Only you can decide what is important to you.
On Thu, 23 May 2024 at 10:04, Bruce Momjian <bruce@momjian.us> wrote: > You might have seen in this thread, I do record commits that speed up > workloads that are user-visible, or specifically make new workloads > possible. I assume that covers the items above, though I have to > determine this from the commit message. It sometimes is hard to write something specific in the commit message about the actual performance increase. For example, if a commit removes an O(N log2 N) algorithm and replaces it with an O(1), you can't say there's an X% increase in performance as the performance increase depends on the value of N. Jelte did call me out for not mentioning enough detail about the performance in c4ab7da60, but if I claimed any % of an increase, it would have been specific to some workload. What is the best way to communicate this stuff so it's easily identifiable when you parse the commit messages? David
On Thu, May 23, 2024 at 01:34:10PM +1200, David Rowley wrote: > On Thu, 23 May 2024 at 10:04, Bruce Momjian <bruce@momjian.us> wrote: > > You might have seen in this thread, I do record commits that speed up > > workloads that are user-visible, or specifically make new workloads > > possible. I assume that covers the items above, though I have to > > determine this from the commit message. > > It sometimes is hard to write something specific in the commit message > about the actual performance increase. > > For example, if a commit removes an O(N log2 N) algorithm and replaces > it with an O(1), you can't say there's an X% increase in performance > as the performance increase depends on the value of N. > > Jelte did call me out for not mentioning enough detail about the > performance in c4ab7da60, but if I claimed any % of an increase, it > would have been specific to some workload. > > What is the best way to communicate this stuff so it's easily > identifiable when you parse the commit messages? This is why I think we need an "Internal Performance" section where we can be clear about simple scaling improvements that might have no user-visible explanation. I would suggest putting it after our "Source code" section. -- Bruce Momjian <bruce@momjian.us> https://momjian.us EDB https://enterprisedb.com Only you can decide what is important to you.
On Thu, 23 May 2024 at 14:01, Bruce Momjian <bruce@momjian.us> wrote: > > On Thu, May 23, 2024 at 01:34:10PM +1200, David Rowley wrote: > > What is the best way to communicate this stuff so it's easily > > identifiable when you parse the commit messages? > > This is why I think we need an "Internal Performance" section where we > can be clear about simple scaling improvements that might have no > user-visible explanation. I would suggest putting it after our "Source > code" section. hmm, but that does not really answer my question. There's still a communication barrier if you're parsing the commit messages are committers don't clearly state some performance improvement numbers for the reason I stated. I also don't agree these should be left to "Source code" section. I feel that section is best suited for extension authors who might care about some internal API change. I'm talking of stuff that makes some user queries possibly N times (!) faster. Surely "Source Code" isn't the place to talk about that? David
Hello,
Regarding this item
: Allow the SLRU cache sizes to be configured (Andrey Borodin, Dilip Kumar)
:
: The new server variables are commit_timestamp_buffers,
: multixact_member_buffers, multixact_offset_buffers, notify_buffers,
: serializable_buffers, subtransaction_buffers, and transaction_buffers.
I hereby request to be listed as third author of this feature.
Also, I'd like to suggest to make it more verbose, as details might be
useful to users. Mention that scalability is improved, because
previously we've suggested to recompile with larger #defines, but to be
cautious because values too high degrade performance. Also mention the
point that some of these grow with shared_buffers is user-visible enough
that it warrants an explicit mention. How about like this:
: Allow the SLRU cache sizes to be configured and improve performance of
: larger caches
: (Andrey Borodin, Dilip Kumar, Álvaro Herrera)
:
: The new server variables are commit_timestamp_buffers,
: multixact_member_buffers, multixact_offset_buffers, notify_buffers,
: serializable_buffers, subtransaction_buffers, and transaction_buffers.
: commit_timestamp_buffers, transaction_buffers and
: subtransaction_buffers scale up automatically with shared_buffers.
These three items
: Allow pg_stat_reset_shared() to reset all shared statistics (Atsushi Torikoshi)
:
: This is done by passing NULL.
:
: Allow pg_stat_reset_shared('slru') to clear SLRU statistics (Atsushi Torikoshi)
:
: Now pg_stat_reset_shared(NULL) also resets SLRU statistics.
:
: Allow pg_stat_reset_slru() to reset all SLRU statistics (Bharath Rupireddy)
:
: The command pg_stat_reset_slru(NULL) already did this.
seem a bit repetitive. (I think the first one is also wrong, because it
says you have to pass NULL, but in reality you can also not give an
argument and it works.) Can we make them a single item? Maybe
something like
: Improve reset routines for shared statistics (Atsushi Torikoshi, Bharath Rupireddy)
:
: Resetting all shared statistics can now be done with
: pg_stat_reset_shared() or pg_stat_reset_shared(NULL), while SLRU
: statistics can now be reset with pg_stat_reset_shared('slru'),
: pg_stat_reset_slru() and pg_stat_reset_slru(NULL).
--
Álvaro Herrera PostgreSQL Developer — https://www.EnterpriseDB.com/
"Find a bug in a program, and fix it, and the program will work today.
Show the program how to find and fix a bug, and the program
will work forever" (Oliver Silfridge)
On Wed, 2024-05-22 at 18:39 -0400, Bruce Momjian wrote:
> On Mon, May 20, 2024 at 11:48:09AM -0700, Jeff Davis wrote:
> > On Sat, 2024-05-18 at 17:51 -0400, Bruce Momjian wrote:
> > > Okay, I went with the attached applied patch. Adjustments?
> >
> > I think it should have more emphasis on the actual new feature: a
> > platform-independent builtin collation provider and the C.UTF-8
> > locale.
> >
> > The user can look at the documentation for comparison with libc.
>
> Okay, changed with the attached applied patch.
Thank you, looks good to me.
Regards,
Jeff Davis
Rename SLRU columns in system view pg_stat_slru (Alvaro Herrera)
The column names accepted by pg_stat_slru_rest() are also changed.
Is pg_stat_slru_rest() correct ?
On Wed, May 22, 2024 at 6:50 PM Bruce Momjian <bruce@momjian.us> wrote: > Agreed, patch applied, thanks. The item for my commit 5bf748b8 currently reads: "Allow btree indexes to more efficiently find array matches" I think that this isn't likely to mean much to most users. It seems like it'd be a lot clearer if the wording was more in line with the beta1 announcement, which talks about the work as an enhancement to index scans that use an IN ( ) condition. Specifically referencing IN() (as opposed to something about arrays or array conditions) is likely to make the item much more understandable. Referencing array matches makes me think of a GIN index on an array column. While ScalarArrayOps do use an array under the cover, that's mostly an implementation detail. At least it is to users that exclusively use IN(), likely the majority (that's the SQL standard syntax). For context, the Postgres 9.2 release notes described the feature that my work directly built on as follows: "Allow indexed_col op ANY(ARRAY[...]) conditions to be used in plain index scans and index-only scans" This was a very accurate description of this earlier work. Similar wording could be used now, but that doesn't seem great to me either. Simply because this wording also doesn't reference IN() conditions in index quals. -- Peter Geoghegan
On Thu, May 23, 2024 at 02:27:07PM +1200, David Rowley wrote: > On Thu, 23 May 2024 at 14:01, Bruce Momjian <bruce@momjian.us> wrote: > > > > On Thu, May 23, 2024 at 01:34:10PM +1200, David Rowley wrote: > > > What is the best way to communicate this stuff so it's easily > > > identifiable when you parse the commit messages? > > > > This is why I think we need an "Internal Performance" section where we > > can be clear about simple scaling improvements that might have no > > user-visible explanation. I would suggest putting it after our "Source > > code" section. > > hmm, but that does not really answer my question. There's still a > communication barrier if you're parsing the commit messages are > committers don't clearly state some performance improvement numbers > for the reason I stated. For a case where O(N^2) become O(N), we might not even know the performance change since it is a micro-optimization. That is why I suggested we call it "Internal Performance". > I also don't agree these should be left to "Source code" section. I > feel that section is best suited for extension authors who might care > about some internal API change. I'm talking of stuff that makes some > user queries possibly N times (!) faster. Surely "Source Code" isn't > the place to talk about that? I said a new section "after our 'Source code' section," not in the source code section. -- Bruce Momjian <bruce@momjian.us> https://momjian.us EDB https://enterprisedb.com Only you can decide what is important to you.
Bruce Momjian <bruce@momjian.us> writes:
> On Thu, May 23, 2024 at 02:27:07PM +1200, David Rowley wrote:
>> I also don't agree these should be left to "Source code" section. I
>> feel that section is best suited for extension authors who might care
>> about some internal API change. I'm talking of stuff that makes some
>> user queries possibly N times (!) faster. Surely "Source Code" isn't
>> the place to talk about that?
> I said a new section "after our 'Source code' section," not in the
> source code section.
Surely, if we make this a separate section, it would come before
'Source code'?
I am not sure Bruce that you realize that your disregard for
performance improvements is shared by nobody. Arguably,
performance is 90% of what we do these days, and it's also
90% of what users care about.
regards, tom lane
On Thu, May 23, 2024 at 11:11:10PM -0400, Tom Lane wrote: > Bruce Momjian <bruce@momjian.us> writes: > > On Thu, May 23, 2024 at 02:27:07PM +1200, David Rowley wrote: > >> I also don't agree these should be left to "Source code" section. I > >> feel that section is best suited for extension authors who might care > >> about some internal API change. I'm talking of stuff that makes some > >> user queries possibly N times (!) faster. Surely "Source Code" isn't > >> the place to talk about that? > > > I said a new section "after our 'Source code' section," not in the > > source code section. > > Surely, if we make this a separate section, it would come before > 'Source code'? > > I am not sure Bruce that you realize that your disregard for > performance improvements is shared by nobody. Arguably, > performance is 90% of what we do these days, and it's also > 90% of what users care about. Please stop saying I don't document performance. I have already explained enough which performance items I choose. Please address my criteria or suggest new criteria. -- Bruce Momjian <bruce@momjian.us> https://momjian.us EDB https://enterprisedb.com Only you can decide what is important to you.
On Thu, May 23, 2024 at 11:04 PM Bruce Momjian <bruce@momjian.us> wrote: > For a case where O(N^2) become O(N), we might not even know the > performance change since it is a micro-optimization. That is why I > suggested we call it "Internal Performance". I don't get this at all. If we can't tell the difference between O(N^2) and O(N), then N was small enough that there wasn't any real need to optimize in the first place. I think we should be assuming that if somebody took the trouble to write a patch, the difference did matter. Hence the change would be user-visible, and should be documented. "Internal Performance" doesn't make a lot of sense to me as a section heading. What does "Internal" mean here as opposed to "General"? I suspect you mean to imply that the user won't be able to tell the difference, but I doubt that very much. We make performance improvements because they are user-visible. If a performance improvement is so miniscule that nobody would ever notice the difference, well then I don't think it needs to be release-noted at all, and we might have a few changes like that where people were mostly aiming for code cleanliness. But in general, what people do is look for something that's slow (for them) and try to make it faster. So the presumption should be that a performance feature has a meaningful impact on users, and then in rare cases we may decide otherwise. -- Robert Haas EDB: http://www.enterprisedb.com
Hi, On 2024-05-23 23:27:04 -0400, Bruce Momjian wrote: > On Thu, May 23, 2024 at 11:11:10PM -0400, Tom Lane wrote: > > Bruce Momjian <bruce@momjian.us> writes: > > I am not sure Bruce that you realize that your disregard for > > performance improvements is shared by nobody. Arguably, > > performance is 90% of what we do these days, and it's also > > 90% of what users care about. > > Please stop saying I don't document performance. I have already > explained enough which performance items I choose. Please address my > criteria or suggest new criteria. Bruce, just about everyone seems to disagree with your current approach. And not just this year, this has been a discussion in most if not all release note threads of the last few years. People, including me, *have* addressed your criteria, but you just waved those concerns away. It's hard to continue discussing criteria when it doesn't at all feel like a conversation. In the end, these are patches to the source code, I don't think you can just wave away widespread disagreement with your changes. That's not how we do postgres development. Greetings, Andres Freund
Hi,
On 2024-05-22 18:33:03 -0400, Bruce Momjian wrote:
> On Tue, May 21, 2024 at 09:40:28AM -0700, Andres Freund wrote:
> > On 2024-05-18 11:13:54 -0400, Bruce Momjian wrote:
> > I agree keeping things reasonably short is important. But I don't think you're
> > evenly applying it as a goal.
> >
> > Just skimming the notes from the end, I see
> > - an 8 entries long pg_stat_statements section
>
> What item did you want to remove? Those are all user-visible changes.
My point here was not that we necessarily need to remove those, but that their
impact to users is smaller than many of the performance impacts you disregard.
> > - multiple entries about "Create custom wait events for ..."
>
> Well, those are all in different sections, so how can they be merged,
> unless I create a "wait event section", I guess.
They're not, all are in "Additional Modules". Instead of
- Create custom wait events for postgres_fdw (Masahiro Ikeda)
- Create custom wait events for dblink (Masahiro Ikeda)
- Allow extensions to define custom wait events (Masahiro Ikeda)
I'd make it:
- Allow extensions to define custom wait events and create custom wait events
for postgres_fdw, dblink (Masahiro Ikeda)
> > - three entries about adding --all to {reindexdb,vacuumdb,clusterdb}.
>
> The problem with merging these is that the "Specifically, --all can now
> be used with" is different for all three of them.
You said you were worried about the length of the release notes, because it
discourages users from actually reading the release notes, due to getting
bored. Having three instance of almost the same entry, with just minor changes
between them, seems to precisely endanger boring readers.
I'd probably just go for
- Add --all option to clusterdb, reindexdb, vacuumdb to process objects in all
databases matching a pattern (Nathan Bossart)
or such. The precise details of how the option works for the different
commands doesn't need to be stated in the release notes, that's more of a
reference documentation thing. But if you want to include it, we can do
something like
Specifically, --all can now be used with --table (all commands), --schema
(reindexdb, vacuumdb), and --exclude-schema (reindexdb, vacuumdb).
> > - an entry about adding long options to pg_archivecleanup
>
> Well, that is a user-visible change. Should it not be listed?
If you are concerned about the length of the release notes and as a
consequence not including more impactful performance changes, then no, it
shouldn't. It doesn't break anyones current scripts, it doesn't enable
anything new.
> > - two entries about grantable maintenance rights, once via pg_maintain, once
> > per-table
>
> Well, one is a GRANT and another is a role, so merging them seemed like
> it would be too confusing.
I don't think it has to be.
Maybe something roughly like
- Allow granting the right to perform maintenance operations (Nathan Bossart)
The permission can be granted on a per-table basis using the MAINTAIN
privilege and on a system wide basis via the pg_maintain role.
Operations that can be controlled are VACUUM, ANALYZE, REINDEX, REFRESH
MATERIALIZED VIEW, CLUSTER, and LOCK TABLE.
I'm again mostly reacting to your concern that the release notes are getting
too boring to read. Repeated content, like in the current formulation, imo
does endanger that. Current it is:
- Add per-table GRANT permission MAINTAIN to control maintenance operations (Nathan Bossart)
The operations are VACUUM, ANALYZE, REINDEX, REFRESH MATERIALIZED VIEW, CLUSTER, and LOCK TABLE.
- Add user-grantable role pg_maintain to control maintenance operations (Nathan Bossart)
The operations are VACUUM, ANALYZE, REINDEX, REFRESH MATERIALIZED VIEW, CLUSTER, and LOCK TABLE.
> > - separate entries about pg_stat_reset_slru(), pg_stat_reset_shared("slru"),
>
> They are different functions with different detail text.
So what? You can change their text. Making it three entries makes it harder
for a reader that doesn't care about resetting stats to skip over the details.
Make it something like
- Improve control over resetting statistics (Atsushi Torikoshi, Bharath
Rupireddy)
pg_stat_reset_shared() can now reset all shared statistics, by passing NULL;
pg_stat_reset_shared(NULL) also resets SLRU statistics;
pg_stat_reset_shared("slru") resets SLRU statistics, which was already
possible using pg_stat_reset_slru(NULL).
Greetings,
Andres Freund
On Fri, May 24, 2024 at 1:50 PM Andres Freund <andres@anarazel.de> wrote: > Bruce, just about everyone seems to disagree with your current approach. And > not just this year, this has been a discussion in most if not all release note > threads of the last few years. +1. > People, including me, *have* addressed your criteria, but you just waved those > concerns away. It's hard to continue discussing criteria when it doesn't at > all feel like a conversation. At one point on this thread, Bruce said "I am particularly critical if I start to wonder, "Why does the author _think_ I should care about this?" because it feels like the author is writing for him/herself and not the audience." Whenever this sort of thing has come up in the past, and I pushed back, Bruce seemed to respond along these lines: he seemed to suggest that there was some kind of conflict of interests involved. This isn't completely unreasonable, of course -- my motivations aren't wholly irrelevant. But for the most part they're *not* very relevant, and wouldn't be even if Bruce's worst suspicions were actually true. In principle it shouldn't matter that I'm biased, if I happen to be correct in some relevant sense. Everybody has some kind of bias. Even if my bias in these matters was a significant factor (which I tend to doubt), I don't think that it's fair to suggest that it's self-serving or careerist. My bias was probably present before I even began work on the feature in question. I tend to work on things because I believe that they're important -- it's not the other way around (at least not to a significant degree). > In the end, these are patches to the source code, I don't think you can just > wave away widespread disagreement with your changes. That's not how we do > postgres development. In lots of cases (a large minority of cases) the problem isn't even really with the emphasis of one type of item over another/the inclusion or non-inclusion of some individual item. It's actually a problem with the information being presented in the most useful way. Often I've suggested what I believe to be a better wording on the merits (usually less obscure and more accessible language), only to be met with the same sort of resistance from Bruce. If I've put a huge amount of work into the item (as is usually the case), then I think that I am at least entitled to a fair hearing. I don't expect Bruce to meet me halfway, or even for him to meet me a quarter of the way -- somebody has to be empowered to say no here (even to very senior community members). I just don't think that he has seriously considered my feedback in this area over the years. Not always, not consistently, but often enough for it to seem like a real problem. -- Peter Geoghegan
On Fri, May 24, 2024 at 10:50:28AM -0700, Andres Freund wrote: > Hi, > > On 2024-05-23 23:27:04 -0400, Bruce Momjian wrote: > > On Thu, May 23, 2024 at 11:11:10PM -0400, Tom Lane wrote: > > > Bruce Momjian <bruce@momjian.us> writes: > > > I am not sure Bruce that you realize that your disregard for > > > performance improvements is shared by nobody. Arguably, > > > performance is 90% of what we do these days, and it's also > > > 90% of what users care about. > > > > Please stop saying I don't document performance. I have already > > explained enough which performance items I choose. Please address my > > criteria or suggest new criteria. > > Bruce, just about everyone seems to disagree with your current approach. And > not just this year, this has been a discussion in most if not all release note > threads of the last few years. > > People, including me, *have* addressed your criteria, but you just waved those > concerns away. It's hard to continue discussing criteria when it doesn't at > all feel like a conversation. > > In the end, these are patches to the source code, I don't think you can just > wave away widespread disagreement with your changes. That's not how we do > postgres development. Well, let's start with a new section for PG 17 that lists these. Is it 20 items, 50, or 150? I have no idea, but without the user-visible filter, I am unable to determine what not-included performance features are worthy of the release notes. Can someone do that? There is no reason other committers can't change the release notes. Yes, I realize we are looking for a consistent voice, but the new section can probably have its own style, and I can make adjustments if desired. Also, I think this has gone unaddressed so long because if we skip a user-visible change, users complain, but I don't remember anyone complaining about skipped performance changes. -- Bruce Momjian <bruce@momjian.us> https://momjian.us EDB https://enterprisedb.com Only you can decide what is important to you.
On Fri, May 24, 2024 at 11:23:29AM -0700, Andres Freund wrote: > Hi, > > On 2024-05-22 18:33:03 -0400, Bruce Momjian wrote: > > On Tue, May 21, 2024 at 09:40:28AM -0700, Andres Freund wrote: > > > On 2024-05-18 11:13:54 -0400, Bruce Momjian wrote: > > > I agree keeping things reasonably short is important. But I don't think you're > > > evenly applying it as a goal. > > > > > > Just skimming the notes from the end, I see > > > - an 8 entries long pg_stat_statements section > > > > What item did you want to remove? Those are all user-visible changes. > > My point here was not that we necessarily need to remove those, but that their > impact to users is smaller than many of the performance impacts you disregard. I liked all your detailed suggestions so applied the attached patch. -- Bruce Momjian <bruce@momjian.us> https://momjian.us EDB https://enterprisedb.com Only you can decide what is important to you.
Attachment
On Thu, May 23, 2024 at 01:22:51PM +0200, Álvaro Herrera wrote:
> Hello,
>
> Regarding this item
>
> : Allow the SLRU cache sizes to be configured (Andrey Borodin, Dilip Kumar)
> :
> : The new server variables are commit_timestamp_buffers,
> : multixact_member_buffers, multixact_offset_buffers, notify_buffers,
> : serializable_buffers, subtransaction_buffers, and transaction_buffers.
>
> I hereby request to be listed as third author of this feature.
>
> Also, I'd like to suggest to make it more verbose, as details might be
> useful to users. Mention that scalability is improved, because
> previously we've suggested to recompile with larger #defines, but to be
> cautious because values too high degrade performance. Also mention the
> point that some of these grow with shared_buffers is user-visible enough
> that it warrants an explicit mention. How about like this:
>
> : Allow the SLRU cache sizes to be configured and improve performance of
> : larger caches
> : (Andrey Borodin, Dilip Kumar, Álvaro Herrera)
> :
> : The new server variables are commit_timestamp_buffers,
> : multixact_member_buffers, multixact_offset_buffers, notify_buffers,
> : serializable_buffers, subtransaction_buffers, and transaction_buffers.
> : commit_timestamp_buffers, transaction_buffers and
> : subtransaction_buffers scale up automatically with shared_buffers.
Yes, I like that, patch applied.
> These three items
>
> : Allow pg_stat_reset_shared() to reset all shared statistics (Atsushi Torikoshi)
> :
> : This is done by passing NULL.
> :
> : Allow pg_stat_reset_shared('slru') to clear SLRU statistics (Atsushi Torikoshi)
> :
> : Now pg_stat_reset_shared(NULL) also resets SLRU statistics.
> :
> : Allow pg_stat_reset_slru() to reset all SLRU statistics (Bharath Rupireddy)
> :
> : The command pg_stat_reset_slru(NULL) already did this.
>
> seem a bit repetitive. (I think the first one is also wrong, because it
> says you have to pass NULL, but in reality you can also not give an
> argument and it works.) Can we make them a single item? Maybe
> something like
>
> : Improve reset routines for shared statistics (Atsushi Torikoshi, Bharath Rupireddy)
> :
> : Resetting all shared statistics can now be done with
> : pg_stat_reset_shared() or pg_stat_reset_shared(NULL), while SLRU
> : statistics can now be reset with pg_stat_reset_shared('slru'),
> : pg_stat_reset_slru() and pg_stat_reset_slru(NULL).
Andres already suggested improvement for this, and I posted the applied
patch. Can you see if that is good or can be improved? Thanks.
--
Bruce Momjian <bruce@momjian.us> https://momjian.us
EDB https://enterprisedb.com
Only you can decide what is important to you.
On Thu, May 23, 2024 at 04:54:28PM -0300, Marcos Pegoraro wrote: > • Rename SLRU columns in system view pg_stat_slru (Alvaro Herrera) > > The column names accepted by pg_stat_slru_rest() are also changed. > > Is pg_stat_slru_rest() correct ? Oops, typo, fixed, thanks. -- Bruce Momjian <bruce@momjian.us> https://momjian.us EDB https://enterprisedb.com Only you can decide what is important to you.
On Thu, May 23, 2024 at 08:19:15PM -0400, Peter Geoghegan wrote:
> On Wed, May 22, 2024 at 6:50 PM Bruce Momjian <bruce@momjian.us> wrote:
> > Agreed, patch applied, thanks.
>
> The item for my commit 5bf748b8 currently reads:
>
> "Allow btree indexes to more efficiently find array matches"
>
> I think that this isn't likely to mean much to most users. It seems
> like it'd be a lot clearer if the wording was more in line with the
> beta1 announcement, which talks about the work as an enhancement to
> index scans that use an IN ( ) condition. Specifically referencing
> IN() (as opposed to something about arrays or array conditions) is
> likely to make the item much more understandable.
>
> Referencing array matches makes me think of a GIN index on an array
> column. While ScalarArrayOps do use an array under the cover, that's
> mostly an implementation detail. At least it is to users that
> exclusively use IN(), likely the majority (that's the SQL standard
> syntax).
>
> For context, the Postgres 9.2 release notes described the feature that
> my work directly built on as follows:
>
> "Allow indexed_col op ANY(ARRAY[...]) conditions to be used in plain
> index scans and index-only scans"
>
> This was a very accurate description of this earlier work. Similar
> wording could be used now, but that doesn't seem great to me either.
> Simply because this wording also doesn't reference IN() conditions in
> index quals.
Agreed. I changed it to:
Allow btree indexes to more efficiently find a set of values, such as
those supplied by IN subqueries
Is that good?
--
Bruce Momjian <bruce@momjian.us> https://momjian.us
EDB https://enterprisedb.com
Only you can decide what is important to you.
On 2024-May-25, Bruce Momjian wrote:
> On Thu, May 23, 2024 at 01:22:51PM +0200, Álvaro Herrera wrote:
> > Can we make them a single item? Maybe something like
> >
> > : Improve reset routines for shared statistics (Atsushi Torikoshi, Bharath Rupireddy)
> > :
> > : Resetting all shared statistics can now be done with
> > : pg_stat_reset_shared() or pg_stat_reset_shared(NULL), while SLRU
> > : statistics can now be reset with pg_stat_reset_shared('slru'),
> > : pg_stat_reset_slru() and pg_stat_reset_slru(NULL).
>
> Andres already suggested improvement for this, and I posted the applied
> patch. Can you see if that is good or can be improved? Thanks.
Yeah, looks good, thanks.
--
Álvaro Herrera 48°01'N 7°57'E — https://www.EnterpriseDB.com/
"No es bueno caminar con un hombre muerto"
On Sun, May 26, 2024 at 10:10:00AM +0200, Álvaro Herrera wrote:
> On 2024-May-25, Bruce Momjian wrote:
>
> > On Thu, May 23, 2024 at 01:22:51PM +0200, Álvaro Herrera wrote:
>
> > > Can we make them a single item? Maybe something like
> > >
> > > : Improve reset routines for shared statistics (Atsushi Torikoshi, Bharath Rupireddy)
> > > :
> > > : Resetting all shared statistics can now be done with
> > > : pg_stat_reset_shared() or pg_stat_reset_shared(NULL), while SLRU
> > > : statistics can now be reset with pg_stat_reset_shared('slru'),
> > > : pg_stat_reset_slru() and pg_stat_reset_slru(NULL).
> >
> > Andres already suggested improvement for this, and I posted the applied
> > patch. Can you see if that is good or can be improved? Thanks.
>
> Yeah, looks good, thanks.
Wow, that's great. My head started to spin trying to make sense of how
those three entries connected. Glad we were able to condense them, and
the new result is easier to read.
--
Bruce Momjian <bruce@momjian.us> https://momjian.us
EDB https://enterprisedb.com
Only you can decide what is important to you.
On Sun, 26 May 2024 at 15:57, Bruce Momjian <bruce@momjian.us> wrote: > Agreed. I changed it to: > > Allow btree indexes to more efficiently find a set of values, such as > those supplied by IN subqueries > > Is that good? I think this needs further adjustment. An "IN subquery" is an IN clause which contains a subquery. As far as I understand it, 5bf748b86 does nothing to improve those. It's there to improve IN with a set of values such as IN(1,2,3). Maybe "IN subqueries" can be replaced with "an SQL IN clause". David
On Tue, May 28, 2024 at 02:44:28PM +1200, David Rowley wrote:
> On Sun, 26 May 2024 at 15:57, Bruce Momjian <bruce@momjian.us> wrote:
> > Agreed. I changed it to:
> >
> > Allow btree indexes to more efficiently find a set of values, such as
> > those supplied by IN subqueries
> >
> > Is that good?
>
> I think this needs further adjustment. An "IN subquery" is an IN
> clause which contains a subquery. As far as I understand it,
> 5bf748b86 does nothing to improve those. It's there to improve IN with
> a set of values such as IN(1,2,3).
>
> Maybe "IN subqueries" can be replaced with "an SQL IN clause".
Okay, I went with:
Allow btree indexes to more efficiently find a set of values,
such as those supplied by IN clauses using constants (Peter Geoghegan,
Matthias van de Meent)
--
Bruce Momjian <bruce@momjian.us> https://momjian.us
EDB https://enterprisedb.com
Only you can decide what is important to you.
On Thu, 9 May 2024 at 05:03, Bruce Momjian <bruce@momjian.us> wrote: > > I have committed the first draft of the PG 17 release notes; you can > see the results here: > > https://momjian.us/pgsql_docs/release-17.html I noticed a couple more things. This item: Add functions to convert integers to hex and binary strings should read: Add functions to convert integers to binary and octal strings The "Improve psql tab completion" item should include this commit: Author: Michael Paquier <michael@paquier.xyz> 2024-05-01 [2800fbb2b] Add tab completion for EXPLAIN (MEMORY|SERIALIZE) and credit Jian He. Regards, Dean
On Wed, Jun 5, 2024 at 11:46:17PM +0100, Dean Rasheed wrote: > On Thu, 9 May 2024 at 05:03, Bruce Momjian <bruce@momjian.us> wrote: > > > > I have committed the first draft of the PG 17 release notes; you can > > see the results here: > > > > https://momjian.us/pgsql_docs/release-17.html > > I noticed a couple more things. This item: > > Add functions to convert integers to hex and binary strings > > should read: > > Add functions to convert integers to binary and octal strings > > > The "Improve psql tab completion" item should include this commit: > > Author: Michael Paquier <michael@paquier.xyz> > 2024-05-01 [2800fbb2b] Add tab completion for EXPLAIN (MEMORY|SERIALIZE) > > and credit Jian He. Agreed, attached patch applied. -- Bruce Momjian <bruce@momjian.us> https://momjian.us EDB https://enterprisedb.com Only you can decide what is important to you.
Attachment
Hi, I noticed that PG17's release note for commit cafe10565 is "Allow psql connections to be canceled with control-C (Tristan Partin)", but this summary seems wrong to me. We already had ^C connection (query) cancellation for quite some time before this patch. What's new with that patch, is that we now also can cancel connection attempts with ^C while we're still connecting (i.e., we haven't yet authenticated and are trying to move the connection state forward). I think a better wording would be "Allow psql connection attempts to be canceled with control-C (Tristan Partin)", or "Allow psql connections to be canceled with control-C while psql is still connecting (Tristan Partin)". Kind regards, Matthias van de Meent Neon (https://neon.tech)
On Fri, Jul 5, 2024 at 07:51:38PM +0200, Matthias van de Meent wrote:
> Hi,
>
> I noticed that PG17's release note for commit cafe10565 is "Allow psql
> connections to be canceled with control-C (Tristan Partin)", but this
> summary seems wrong to me.
>
> We already had ^C connection (query) cancellation for quite some time
> before this patch. What's new with that patch, is that we now also can
> cancel connection attempts with ^C while we're still connecting (i.e.,
> we haven't yet authenticated and are trying to move the connection
> state forward).
> I think a better wording would be "Allow psql connection attempts to
> be canceled with control-C (Tristan Partin)", or "Allow psql
> connections to be canceled with control-C while psql is still
> connecting (Tristan Partin)".
I see your point. I committed a change to use this wording:
Allow psql connection attempts to be canceled with control-C
(Tristan Partin)
--
Bruce Momjian <bruce@momjian.us> https://momjian.us
EDB https://enterprisedb.com
Only you can decide what is important to you.
Hi,
In the PG17 release notes, I noticed it is mentioned as "pg_attribute.stxstattarget" which seems incorrect.
In my opinion, it should be "pg_statistic_ext.stxstattarget" because the "stxstattarget" column is part of the "pg_statistic_ext" catalog.
Regards,
Kisoon Kwon
Bitnine Global (www.bitnine.net)
2024년 7월 6일 (토) 오전 5:53, Bruce Momjian <bruce@momjian.us>님이 작성:
On Fri, Jul 5, 2024 at 07:51:38PM +0200, Matthias van de Meent wrote:
> Hi,
>
> I noticed that PG17's release note for commit cafe10565 is "Allow psql
> connections to be canceled with control-C (Tristan Partin)", but this
> summary seems wrong to me.
>
> We already had ^C connection (query) cancellation for quite some time
> before this patch. What's new with that patch, is that we now also can
> cancel connection attempts with ^C while we're still connecting (i.e.,
> we haven't yet authenticated and are trying to move the connection
> state forward).
> I think a better wording would be "Allow psql connection attempts to
> be canceled with control-C (Tristan Partin)", or "Allow psql
> connections to be canceled with control-C while psql is still
> connecting (Tristan Partin)".
I see your point. I committed a change to use this wording:
Allow psql connection attempts to be canceled with control-C
(Tristan Partin)
--
Bruce Momjian <bruce@momjian.us> https://momjian.us
EDB https://enterprisedb.com
Only you can decide what is important to you.
Hi, On Thu, 9 May 2024 00:03:50 -0400 Bruce Momjian <bruce@momjian.us> wrote: > I have committed the first draft of the PG 17 release notes; you can > see the results here: > > https://momjian.us/pgsql_docs/release-17.html > > It will be improved until the final release. The item count is 188, > which is similar to recent releases: > > release-10: 189 > release-11: 170 > release-12: 180 > release-13: 178 > release-14: 220 > release-15: 184 > release-16: 206 > release-17: 188 > > I welcome feedback. For some reason it was an easier job than usual. I found the following in the release notes: Change file boundary handling of two WAL file name functions (Kyotaro Horiguchi, Andres Freund, Bruce Momjian) The functions pg_walfile_name() and pg_walfile_name_offset() used to report the previous LSN segment number when the LSN was on a file segment boundary; it now returns the LSN segment. It might be trivial, but, reading the associated commit message , I think it would be more explicit for users to rewrite the last statement to "it now returns the current LSN segment." Regards, Yugo Nagata > > -- > Bruce Momjian <bruce@momjian.us> https://momjian.us > EDB https://enterprisedb.com > > Only you can decide what is important to you. > > -- Yugo Nagata <nagata@sraoss.co.jp>
> Add server variable huge_page_size to report the use of huge pages by The new variable is huge_page_status; h_p_size is several years old. BTW, I was surprised that these were included: +2024-02-28 [363eb0599] Convert README to Markdown. +2024-01-25 [7014c9a4b] Doc: improve documentation for jsonpath behavior. -- Justin
On Thu, Aug 8, 2024 at 08:55:53AM -0500, Justin Pryzby wrote: > > Add server variable huge_page_size to report the use of huge pages by > > The new variable is huge_page_status; h_p_size is several years old. Fixed. I created this mistake when I was adding links to the SGML file. > BTW, I was surprised that these were included: > > +2024-02-28 [363eb0599] Convert README to Markdown. > +2024-01-25 [7014c9a4b] Doc: improve documentation for jsonpath behavior. I try to mention significant doc changes, and have done so in the past. -- Bruce Momjian <bruce@momjian.us> https://momjian.us EDB https://enterprisedb.com Only you can decide what is important to you.
While freely acknowledging that I am biased because I wrote it, I am a bit surprised to see the DSM registry left out of the release notes (commit 8b2bcf3, docs are here [0]). This feature is intended to allow modules to allocate shared memory after startup, i.e., without requiring the module to be loaded via shared_preload_libraries. IMHO that is worth mentioning. [0] https://www.postgresql.org/docs/devel/xfunc-c.html#XFUNC-SHARED-ADDIN-AFTER-STARTUP -- nathan
hi. Allow partitions to be merged using ALTER TABLE ... MERGE PARTITIONS (Dmitry Koval) Allow partitions to be split using ALTER TABLE ... SPLIT PARTITION (Dmitry Koval) also these two items got reverted? see https://git.postgresql.org/cgit/postgresql.git/commit/?id=3890d90c1508125729ed20038d90513694fc3a7b
On Wed, Sep 4, 2024 at 07:18:52PM +0800, jian he wrote: > hi. > > Allow partitions to be merged using ALTER TABLE ... MERGE PARTITIONS > (Dmitry Koval) > Allow partitions to be split using ALTER TABLE ... SPLIT PARTITION > (Dmitry Koval) > > also these two items got reverted? see > https://git.postgresql.org/cgit/postgresql.git/commit/?id=3890d90c1508125729ed20038d90513694fc3a7b I don't see them in the PG 17 release notes at: https://www.postgresql.org/docs/17/release-17.html I did just remove the tab complete comment for this though. -- Bruce Momjian <bruce@momjian.us> https://momjian.us EDB https://enterprisedb.com Only you can decide what is important to you.
On Tue, Sep 3, 2024 at 10:44:01AM -0500, Nathan Bossart wrote: > While freely acknowledging that I am biased because I wrote it, I am a bit > surprised to see the DSM registry left out of the release notes (commit > 8b2bcf3, docs are here [0]). This feature is intended to allow modules to > allocate shared memory after startup, i.e., without requiring the module to > be loaded via shared_preload_libraries. IMHO that is worth mentioning. > > [0] https://www.postgresql.org/docs/devel/xfunc-c.html#XFUNC-SHARED-ADDIN-AFTER-STARTUP That seems more infrastructure/extension author stuff which isn't normally mentioned in the release notes. I think such people really need to look at all the commit messages. -- Bruce Momjian <bruce@momjian.us> https://momjian.us EDB https://enterprisedb.com Only you can decide what is important to you.
On 2024-Sep-05, Bruce Momjian wrote: > That seems more infrastructure/extension author stuff which isn't > normally mentioned in the release notes. I think such people really > need to look at all the commit messages. Are you saying all extension authors should be reading the complete git log for every single major release? That's a strange position to take. Isn't this a good fit for "E.1.3.10. Source Code"? -- Álvaro Herrera 48°01'N 7°57'E — https://www.EnterpriseDB.com/ <Schwern> It does it in a really, really complicated way <crab> why does it need to be complicated? <Schwern> Because it's MakeMaker.
On Sat, Sep 7, 2024 at 11:55:09AM +0200, Álvaro Herrera wrote: > On 2024-Sep-05, Bruce Momjian wrote: > > > That seems more infrastructure/extension author stuff which isn't > > normally mentioned in the release notes. I think such people really > > need to look at all the commit messages. > > Are you saying all extension authors should be reading the complete git > log for every single major release? That's a strange position to take. > Isn't this a good fit for "E.1.3.10. Source Code"? Yes. There are so many changes at the source code level it is unwise to try and get them into the main release notes. If someone wants to create an addendum, like was suggested for pure performance improvements, that would make sense. -- Bruce Momjian <bruce@momjian.us> https://momjian.us EDB https://enterprisedb.com Only you can decide what is important to you.
On Tue, 10 Sept 2024 at 04:47, Bruce Momjian <bruce@momjian.us> wrote: > Yes. There are so many changes at the source code level it is unwise to > try and get them into the main release notes. If someone wants to > create an addendum, like was suggested for pure performance > improvements, that would make sense. I agree that the release notes cannot fit every change. But I also don't think any extension author reads the complete git commit log every release, so taking the stance that they should be seems unhelpful. And the "Source Code" section does exist so at some level you seem to disagree with that too. So what is the way to decide that something makes the cut for the "Source Code" section? I think as an extension author there are usually three types of changes that are relevant: 1. New APIs/hooks that are meant for extension authors 2. Stuff that causes my existing code to not compile anymore 3. Stuff that changes behaviour of existing APIs code in a incompatible but silent way For 1, I think adding them to the release notes makes total sense, especially if the new APIs are documented not only in source code, but also on the website. Nathan his change is of this type, so I agree with him it should be in the release notes. For 2, I'll be able to easily find the PG commit that caused the compilation failure by grepping git history for the old API. So having these changes in the release notes seems unnecessary. For 3, it would be very useful if it would be in the release notes, but I think in many cases it's hard to know what commits do this. So unless it's obviously going to break a bunch of extensions silently, I think we don't have to add such changes to the release notes.
On Thu, Sep 05, 2024 at 09:51:25PM -0400, Bruce Momjian wrote: > On Tue, Sep 3, 2024 at 10:44:01AM -0500, Nathan Bossart wrote: > > While freely acknowledging that I am biased because I wrote it, I am a bit > > surprised to see the DSM registry left out of the release notes (commit > > 8b2bcf3, docs are here [0]). This feature is intended to allow modules to > > allocate shared memory after startup, i.e., without requiring the module to > > be loaded via shared_preload_libraries. IMHO that is worth mentioning. > > > > [0] https://www.postgresql.org/docs/devel/xfunc-c.html#XFUNC-SHARED-ADDIN-AFTER-STARTUP > > That seems more infrastructure/extension author stuff which isn't > normally mentioned in the release notes. If I understand the feature correctly, it allows extensions to be just CREATEd without having them to be added to shared_preload_libraries, i.e. saving the organization an instance restart/downtime. That seems important enough for end-users to know, even if they will need to wait for extension authors to catch up to this (but I guess a lot will). Michael
On 2024-Sep-10, Jelte Fennema-Nio wrote: > I think as an extension author there are usually three types of > changes that are relevant: > > 1. New APIs/hooks that are meant for extension authors > For 1, I think adding them to the release notes makes total sense, > especially if the new APIs are documented not only in source code, but > also on the website. Nathan his change is of this type, so I agree > with him it should be in the release notes. I agree. The volume of such items should be pretty small. > 3. Stuff that changes behaviour of existing APIs code in a > incompatible but silent way > For 3, it would be very useful if it would be in the release notes, > but I think in many cases it's hard to know what commits do this. So > unless it's obviously going to break a bunch of extensions silently, I > think we don't have to add such changes to the release notes. While we cannot be 100% vigilant (and it doesn't seem likely for automated tools to detect this), we try to avoid API changes that would still compile but behave incompatibly. In many review discussions you can see suggestions to change some function signature so that third-party authors would be aware that they need to adapt their code to new behavior, turning cases of (3) into (2). I agree that these don't need release notes items. -- Álvaro Herrera PostgreSQL Developer — https://www.EnterpriseDB.com/ "XML!" Exclaimed C++. "What are you doing here? You're not a programming language." "Tell that to the people who use me," said XML. https://burningbird.net/the-parable-of-the-languages/
On Mon, Sep 9, 2024 at 11:29 PM Jelte Fennema-Nio <postgres@jeltef.nl> wrote: > > On Tue, 10 Sept 2024 at 04:47, Bruce Momjian <bruce@momjian.us> wrote: > > Yes. There are so many changes at the source code level it is unwise to > > try and get them into the main release notes. If someone wants to > > create an addendum, like was suggested for pure performance > > improvements, that would make sense. > > I agree that the release notes cannot fit every change. But I also > don't think any extension author reads the complete git commit log > every release, so taking the stance that they should be seems > unhelpful. And the "Source Code" section does exist so at some level > you seem to disagree with that too. So what is the way to decide that > something makes the cut for the "Source Code" section? > > I think as an extension author there are usually three types of > changes that are relevant: > 1. New APIs/hooks that are meant for extension authors > 2. Stuff that causes my existing code to not compile anymore > 3. Stuff that changes behaviour of existing APIs code in a > incompatible but silent way > > For 1, I think adding them to the release notes makes total sense, > especially if the new APIs are documented not only in source code, but > also on the website. Nathan his change is of this type, so I agree > with him it should be in the release notes. +1. I think that the increment JSON parser that is already mentioned in the release note would fall in this type too; it's not a feature aimed just for extension authors, but it's kind of source and internal changes IMO. Since the DSM registry feature is described in the doc, I think it would make sense to have it in the release notes and probably has a link to the "Requesting Shared Memory After Startup" section. Regards, -- Masahiko Sawada Amazon Web Services: https://aws.amazon.com
On Tue, Sep 10, 2024 at 08:28:42AM +0200, Jelte Fennema-Nio wrote:
> I think as an extension author there are usually three types of
> changes that are relevant:
> 1. New APIs/hooks that are meant for extension authors
> 2. Stuff that causes my existing code to not compile anymore
> 3. Stuff that changes behaviour of existing APIs code in a
> incompatible but silent way
>
> For 1, I think adding them to the release notes makes total sense,
> especially if the new APIs are documented not only in source code, but
> also on the website. Nathan his change is of this type, so I agree
> with him it should be in the release notes.
>
> For 2, I'll be able to easily find the PG commit that caused the
> compilation failure by grepping git history for the old API. So having
> these changes in the release notes seems unnecessary.
>
> For 3, it would be very useful if it would be in the release notes,
> but I think in many cases it's hard to know what commits do this. So
> unless it's obviously going to break a bunch of extensions silently, I
> think we don't have to add such changes to the release notes.
So, we are looking at this commit:
commit b5a9b18cd0b
Author: Thomas Munro <tmunro@postgresql.org>
Date: Wed Apr 3 00:17:06 2024 +1300
Provide API for streaming relation data.
Introduce an abstraction allowing relation data to be accessed as a
stream of buffers, with an implementation that is more efficient than
the equivalent sequence of ReadBuffer() calls.
Client code supplies a callback that can say which block number it wants
next, and then consumes individual buffers one at a time from the
stream. This division puts read_stream.c in control of how far ahead it
can see and allows it to read clusters of neighboring blocks with
StartReadBuffers(). It also issues POSIX_FADV_WILLNEED advice ahead of
time when random access is detected.
Other variants of I/O stream will be proposed in future work (for
example to support recovery, whose LsnReadQueue device in
xlogprefetcher.c is a distant cousin of this code and should eventually
be replaced by this), but this basic API is sufficient for many common
executor usage patterns involving predictable access to a single fork of
a single relation.
Several patches using this API are proposed separately.
This stream concept is loosely based on ideas from Andres Freund on how
we should pave the way for later work on asynchronous I/O.
You are right that I do mention changes specifically designed for the
use of extensions, but there is no mention in the commit message of its
use for extensions. In fact, I thought this was too low-level to be of
use for extensions. However, if people feel it should be added, we have
enough time to add it.
I also mention changes that are _likely_ to affect extensions, but not
all changes that could affect extensions. An interesting idea would be
to report all function signature changes in each major release in some
way.
--
Bruce Momjian <bruce@momjian.us> https://momjian.us
EDB https://enterprisedb.com
When a patient asks the doctor, "Am I going to die?", he means
"Am I going to die soon?"
On Tue, Sep 10, 2024 at 09:52:42AM -0700, Masahiko Sawada wrote:
> On Mon, Sep 9, 2024 at 11:29 PM Jelte Fennema-Nio <postgres@jeltef.nl> wrote:
> >
> > On Tue, 10 Sept 2024 at 04:47, Bruce Momjian <bruce@momjian.us> wrote:
> > > Yes. There are so many changes at the source code level it is unwise to
> > > try and get them into the main release notes. If someone wants to
> > > create an addendum, like was suggested for pure performance
> > > improvements, that would make sense.
> >
> > I agree that the release notes cannot fit every change. But I also
> > don't think any extension author reads the complete git commit log
> > every release, so taking the stance that they should be seems
> > unhelpful. And the "Source Code" section does exist so at some level
> > you seem to disagree with that too. So what is the way to decide that
> > something makes the cut for the "Source Code" section?
> >
> > I think as an extension author there are usually three types of
> > changes that are relevant:
> > 1. New APIs/hooks that are meant for extension authors
> > 2. Stuff that causes my existing code to not compile anymore
> > 3. Stuff that changes behaviour of existing APIs code in a
> > incompatible but silent way
> >
> > For 1, I think adding them to the release notes makes total sense,
> > especially if the new APIs are documented not only in source code, but
> > also on the website. Nathan his change is of this type, so I agree
> > with him it should be in the release notes.
>
> +1. I think that the increment JSON parser that is already mentioned
> in the release note would fall in this type too; it's not a feature
> aimed just for extension authors, but it's kind of source and internal
> changes IMO. Since the DSM registry feature is described in the doc, I
> think it would make sense to have it in the release notes and probably
> has a link to the "Requesting Shared Memory After Startup" section.
This commit?
commit 8b2bcf3f287
Author: Nathan Bossart <nathan@postgresql.org>
Date: Fri Jan 19 14:24:36 2024 -0600
Introduce the dynamic shared memory registry.
Yes, we have time to add it.
--
Bruce Momjian <bruce@momjian.us> https://momjian.us
EDB https://enterprisedb.com
When a patient asks the doctor, "Am I going to die?", he means
"Am I going to die soon?"
On Wed, Sep 11, 2024 at 10:12:58AM -0400, Bruce Momjian wrote: > On Tue, Sep 10, 2024 at 09:52:42AM -0700, Masahiko Sawada wrote: >> On Mon, Sep 9, 2024 at 11:29 PM Jelte Fennema-Nio <postgres@jeltef.nl> wrote: >> > For 1, I think adding them to the release notes makes total sense, >> > especially if the new APIs are documented not only in source code, but >> > also on the website. Nathan his change is of this type, so I agree >> > with him it should be in the release notes. >> >> +1. I think that the increment JSON parser that is already mentioned >> in the release note would fall in this type too; it's not a feature >> aimed just for extension authors, but it's kind of source and internal >> changes IMO. Since the DSM registry feature is described in the doc, I >> think it would make sense to have it in the release notes and probably >> has a link to the "Requesting Shared Memory After Startup" section. > > This commit? > > commit 8b2bcf3f287 > Author: Nathan Bossart <nathan@postgresql.org> > Date: Fri Jan 19 14:24:36 2024 -0600 > > Introduce the dynamic shared memory registry. > > Yes, we have time to add it. Yes, that's the one. -- nathan
Hello,
I noticed that these two items in the current notes are separate:
<!--
Author: Alvaro Herrera <alvherre@alvh.no-ip.org>
2024-03-25 [374c7a229] Allow specifying an access method for partitioned tables
Author: Alvaro Herrera <alvherre@alvh.no-ip.org>
2024-03-28 [e2395cdbe] ALTER TABLE: rework determination of access method ID
-->
<listitem>
<para>
Allow specification of partitioned <link linkend="tableam">table
access methods</link> (Justin Pryzby, Soumyadeep Chakraborty,
Michael Paquier)
</para>
</listitem>
<!--
Author: Michael Paquier <michael@paquier.xyz>
2024-03-08 [d61a6cad6] Add support for DEFAULT in ALTER TABLE .. SET ACCESS MET
-->
<listitem>
<para>
Add <literal>DEFAULT</literal> setting for <literal>ALTER TABLE
.. SET ACCESS METHOD</literal> (Michael Paquier)
</para>
</listitem>
They are very very closely related, so I suggest they should be
together as a single item. Also, the first one is somewhat strangely
worded IMO (we don't have "partitioned table access methods" -- rather,
we have table access methods for partitioned tables). Maybe something
like
* Improve ALTER TABLE ... SET ACCESS METHOD
This command can now also be applied to partitioned tables, so that it
can <link to "https://www.postgresql.org/docs/17/sql-altertable.html#SQL-ALTERTABLE-DESC-SET-ACCESS-METHOD">
influence partitions created later</link>. (Justin, Soumyadeep, Michaël)
In addition, it now accepts the value DEFAULT to reset a previously
set value. (Michaël)
There's also a bunch of items on EXPLAIN, which could perhaps be grouped
in a single item with sub-paras for each individual change; I'd also
move it to the bottom of E.1.3.2.
--
Álvaro Herrera Breisgau, Deutschland — https://www.EnterpriseDB.com/
"This is a foot just waiting to be shot" (Andrew Dunstan)
On Fri, Sep 13, 2024 at 04:17:31PM -0400, Bruce Momjian wrote: > Attached patch applied, with commit URL link. Looks good, thanks. -- nathan
On Wed, Sep 11, 2024 at 07:50:40PM +0200, Álvaro Herrera wrote:
> There's also a bunch of items on EXPLAIN, which could perhaps be grouped
> in a single item with sub-paras for each individual change; I'd also
> move it to the bottom of E.1.3.2.
Oh, I hadn't noticed I have five EXPLAIN items --- that is enough to
make a new section, done at:
https://momjian.us/pgsql_docs/release-17.html#RELEASE-17-EXPLAIN
I don't think I can combine the EXPLAIN items without making them too
complex.
--
Bruce Momjian <bruce@momjian.us> https://momjian.us
EDB https://enterprisedb.com
When a patient asks the doctor, "Am I going to die?", he means
"Am I going to die soon?"
On Wed, 11 Sept 2024 at 16:10, Bruce Momjian <bruce@momjian.us> wrote:
> You are right that I do mention changes specifically designed for the
> use of extensions, but there is no mention in the commit message of its
> use for extensions. In fact, I thought this was too low-level to be of
> use for extensions. However, if people feel it should be added, we have
> enough time to add it.
Another new API that is useful for extension authors is the following
one (I'm obviously biased since I'm the author, and I don't know if
there's still time):
commit 14dd0f27d7cd56ffae9ecdbe324965073d01a9ff
Author: Nathan Bossart <nathan@postgresql.org>
Date: Thu Jan 4 16:09:34 2024 -0600
Add macros for looping through a List without a ListCell.
Many foreach loops only use the ListCell pointer to retrieve the
content of the cell, like so:
ListCell *lc;
foreach(lc, mylist)
{
int myint = lfirst_int(lc);
...
}
This commit adds a few convenience macros that automatically
declare the loop variable and retrieve the current cell's contents.
This allows us to rewrite the previous loop like this:
foreach_int(myint, mylist)
{
...
}
> An interesting idea would be
> to report all function signature changes in each major release in some
> way.
I think that might be useful, but it very much depends how long that
list gets. If it gets too long I think authors will just try to
compile and only look at the ones that break for them.
On 2024-Sep-11, Bruce Momjian wrote: > An interesting idea would be to report all function signature changes > in each major release in some way. Hmm, extension authors are going to realize this as soon as they try to compile, so it doesn't seem necessary. Having useful APIs _added_ is a different matter, because those might help them realize that they can remove parts (or #ifdef-out for newer PG versions) of their code, or add new features; there's no Clippit saying "it looks like you're compiling for Postgres 18, would you like to ...?". -- Álvaro Herrera Breisgau, Deutschland — https://www.EnterpriseDB.com/ "The important things in the world are problems with society that we don't understand at all. The machines will become more complicated but they won't be more complicated than the societies that run them." (Freeman Dyson)
On Tue, Sep 17, 2024 at 10:01:28AM +0200, Jelte Fennema-Nio wrote:
> On Wed, 11 Sept 2024 at 16:10, Bruce Momjian <bruce@momjian.us> wrote:
> > You are right that I do mention changes specifically designed for the
> > use of extensions, but there is no mention in the commit message of its
> > use for extensions. In fact, I thought this was too low-level to be of
> > use for extensions. However, if people feel it should be added, we have
> > enough time to add it.
>
> Another new API that is useful for extension authors is the following
> one (I'm obviously biased since I'm the author, and I don't know if
> there's still time):
>
> commit 14dd0f27d7cd56ffae9ecdbe324965073d01a9ff
> Author: Nathan Bossart <nathan@postgresql.org>
> Date: Thu Jan 4 16:09:34 2024 -0600
>
> Add macros for looping through a List without a ListCell.
>
> Many foreach loops only use the ListCell pointer to retrieve the
> content of the cell, like so:
>
> ListCell *lc;
>
> foreach(lc, mylist)
> {
> int myint = lfirst_int(lc);
>
> ...
> }
>
> This commit adds a few convenience macros that automatically
> declare the loop variable and retrieve the current cell's contents.
> This allows us to rewrite the previous loop like this:
>
> foreach_int(myint, mylist)
> {
> ...
> }
Can someone else comment on the idea of adding this release note item?
I don't feel confident in my ability to evaluate this. I obviously did
not see it as significant the first time.
--
Bruce Momjian <bruce@momjian.us> https://momjian.us
EDB https://enterprisedb.com
When a patient asks the doctor, "Am I going to die?", he means
"Am I going to die soon?"
On Wed, Sep 18, 2024 at 05:33:18PM -0400, Bruce Momjian wrote: > On Tue, Sep 17, 2024 at 10:01:28AM +0200, Jelte Fennema-Nio wrote: >> Another new API that is useful for extension authors is the following >> one (I'm obviously biased since I'm the author, and I don't know if >> there's still time): >> >> Add macros for looping through a List without a ListCell. > > Can someone else comment on the idea of adding this release note item? > I don't feel confident in my ability to evaluate this. I obviously did > not see it as significant the first time. I'm not sure precisely what criteria you use to choose what goes in the release notes, but this one seems like a judgement call to me. My initial reaction is that it shouldn't be included, but I do see some items with a similar scope, such as "Remove some SPI macros." -- nathan
On Wed, Sep 18, 2024 at 9:09 PM Nathan Bossart <nathandbossart@gmail.com> wrote: > On Wed, Sep 18, 2024 at 05:33:18PM -0400, Bruce Momjian wrote: > > On Tue, Sep 17, 2024 at 10:01:28AM +0200, Jelte Fennema-Nio wrote: > >> Another new API that is useful for extension authors is the following > >> one (I'm obviously biased since I'm the author, and I don't know if > >> there's still time): > >> > >> Add macros for looping through a List without a ListCell. > > > > Can someone else comment on the idea of adding this release note item? > > I don't feel confident in my ability to evaluate this. I obviously did > > not see it as significant the first time. > > I'm not sure precisely what criteria you use to choose what goes in the > release notes, but this one seems like a judgement call to me. My initial > reaction is that it shouldn't be included, but I do see some items with a > similar scope, such as "Remove some SPI macros." I wouldn't mention either this or "Remove some unused SPI macros". -- Robert Haas EDB: http://www.enterprisedb.com
On Thu, Sep 19, 2024 at 12:23:21PM -0400, Robert Haas wrote: > On Wed, Sep 18, 2024 at 9:09 PM Nathan Bossart <nathandbossart@gmail.com> wrote: > > On Wed, Sep 18, 2024 at 05:33:18PM -0400, Bruce Momjian wrote: > > > On Tue, Sep 17, 2024 at 10:01:28AM +0200, Jelte Fennema-Nio wrote: > > >> Another new API that is useful for extension authors is the following > > >> one (I'm obviously biased since I'm the author, and I don't know if > > >> there's still time): > > >> > > >> Add macros for looping through a List without a ListCell. > > > > > > Can someone else comment on the idea of adding this release note item? > > > I don't feel confident in my ability to evaluate this. I obviously did > > > not see it as significant the first time. > > > > I'm not sure precisely what criteria you use to choose what goes in the > > release notes, but this one seems like a judgement call to me. My initial > > reaction is that it shouldn't be included, but I do see some items with a > > similar scope, such as "Remove some SPI macros." > > I wouldn't mention either this or "Remove some unused SPI macros". I mentioned the SPI macros because that could lead to breakage, and there might be applications, as well as extensions, that use it. -- Bruce Momjian <bruce@momjian.us> https://momjian.us EDB https://enterprisedb.com When a patient asks the doctor, "Am I going to die?", he means "Am I going to die soon?"
On Thu, Sep 19, 2024 at 1:04 PM Bruce Momjian <bruce@momjian.us> wrote: > I mentioned the SPI macros because that could lead to breakage, and > there might be applications, as well as extensions, that use it. Sure, this is all a judgement call. I don't think it's particularly likely that many people are relying on those macros, though, and if they are, they will mostly likely find out that they're gone when they try to compile, rather than from reading the release notes. Likewise, I feel that the new list iteration macros are both optional and minor, so there's not really a reason to tell people about them. But opinions will vary, and that's fine. I just mentioned my opinion since you seemed to be asking. :-) -- Robert Haas EDB: http://www.enterprisedb.com
On Fri, 2024-09-20 at 10:02 -0400, Jonathan S. Katz wrote: > Attached is a proposal for the major features section. This borrows from > the release announcement draft[1] and lists out features and themes that > have broad user impact. This was a bit challenging for this release, > because there are a lot of great features in PG17 that add up to a very > special release. > > Feedback welcome. I would have added the platform-independent binary collation provider. And perhaps "pg_createsubscriber": that can be a game-changer for setting up logical replication. Yours, Laurenz Albe
On Fri, Sep 20, 2024 at 01:47:43PM -0400, Jonathan Katz wrote: > On 9/20/24 12:55 PM, Laurenz Albe wrote: > > On Fri, 2024-09-20 at 10:02 -0400, Jonathan S. Katz wrote: > > > Attached is a proposal for the major features section. This borrows from > > > the release announcement draft[1] and lists out features and themes that > > > have broad user impact. This was a bit challenging for this release, > > > because there are a lot of great features in PG17 that add up to a very > > > special release. > > > > > > Feedback welcome. > > > > I would have added the platform-independent binary collation provider. > > And perhaps "pg_createsubscriber": that can be a game-changer for setting > > up logical replication. > > I was on the fence about that, mostly because it'd make that sentence too > much of a mouthful, but I do agree. > > IIRC (didn't get to check) we did have a precedent for sublists in the major > features, so I broke this one up. Please see attached. Patch applied to PG 17. -- Bruce Momjian <bruce@momjian.us> https://momjian.us EDB https://enterprisedb.com When a patient asks the doctor, "Am I going to die?", he means "Am I going to die soon?"
Bruce Momjian <bruce@momjian.us> writes:
> Patch applied to PG 17.
I don't see a push?
regards, tom lane
On Fri, 2024-09-20 at 13:47 -0400, Jonathan S. Katz wrote: > Please see attached. > + <listitem> > + <para> > + Various query performance improvements, including to sequential reads > + using streaming I/O, write throughput under high concurrency, and > + searches over multiple values in a <link linkend="btree">btree</link> > + index. > + </para> > + </listitem> Perhaps that last part could be "and searches over IN-lists in a b-tree index". It might be technically less correct, but I'd expect that it gives more people the right idea. > + <para> > + <link > + linkend="app-pgcreatesubscriber"><application>pg_createsubscriber</application></link>, > + a utility that logical replicas from physical standbys > + </para> There's a verb missing: "a utility that *creates* logical replicas..." > + <para> > + <link > + linkend="pgupgrade"><application>pg_upgrade</application></link> now > + preserves replication slots on both publishers and subscribers > + </para> I wonder if we should omit "on both publishers and subscribers". It preserves replication slots anywhere, right? > + <listitem> > + <para> > + New client-side connection option, <link > + linkend="libpq-connect-sslnegotiation"><literal>sslnegotiation=direct</literal></link>, > + that allows direct TLS handshakes that avoids a round-trip negotation. > + </para> > + </listitem> It should be "that avoid". Plural. Yours, Laurenz Albe
On Fri, Sep 20, 2024 at 04:05:11PM -0400, Tom Lane wrote: > Bruce Momjian <bruce@momjian.us> writes: > > Patch applied to PG 17. > > I don't see a push? Push was delayed because my test script found some uncommitted files due to earlier testing. Should be fine now. -- Bruce Momjian <bruce@momjian.us> https://momjian.us EDB https://enterprisedb.com When a patient asks the doctor, "Am I going to die?", he means "Am I going to die soon?"
Laurenz Albe <laurenz.albe@cybertec.at> writes:
> [ assorted corrections ]
I fixed a couple of these before seeing your message.
regards, tom lane
On Sat, Sep 21, 2024 at 1:50 AM Bruce Momjian <bruce@momjian.us> wrote:
>
> On Fri, Sep 20, 2024 at 04:05:11PM -0400, Tom Lane wrote:
> > Bruce Momjian <bruce@momjian.us> writes:
> > > Patch applied to PG 17.
> >
> > I don't see a push?
>
> Push was delayed because my test script found some uncommitted files due
> to earlier testing. Should be fine now.
>
<para>
<link
linkend="app-pgcreatesubscriber"><application>pg_createsubscriber</application></link>,
a utility that creates logical replicas from physical standbys
</para>
This description is okay but according to me, the more compelling use
case is that this new utility helps to allow online upgrades of
physical replication setup as explained in the blog [1]. See the
section: "Upgrading Streaming (Physical) Replication Setup".
</listitem>
<listitem>
<para>
<link
linkend="pgupgrade"><application>pg_upgrade</application></link> now
preserves replication slots on both publishers and subscribers
</para>
It is better to write the above statement as:
"pg_upgrade</application></link> now preserves replication slots on
publishers and full subscription's state on subscribers". This is
because replication slots are preserved on publishers. The subscribers
preserve the subscription state.
[1] - http://amitkapila16.blogspot.com/2024/09/online-upgrading-logical-and-physical.html
--
With Regards,
Amit Kapila.
On Thu, Sep 26, 2024 at 02:19:21PM +0900, Yugo Nagata wrote: > On Thu, 9 May 2024 00:03:50 -0400 > Bruce Momjian <bruce@momjian.us> wrote: > > > I have committed the first draft of the PG 17 release notes; you can > > see the results here: > > I propose to improve the following description in "Migration to Version 17" > section by adding CREATE INDEX and CREATE MATERIALIZED VIEW into the command list. > > <para> > Change functions to use a safe <xref linkend="guc-search-path"/> > during maintenance operations (Jeff Davis) > <ulink url="&commit_baseurl;2af07e2f7">§</ulink> > </para> > > It is suggested in the thread [1] that users could not notice the behaviour > of CREATE INDEX is changed because the explicit command name is not listed in > the release notes. So, I think it is better to add CREATE INDEX and > CREATE MATERIALIZED VIEW into the command list. > > I've attached a patch. It this a valid change? Seems so. -- Bruce Momjian <bruce@momjian.us> https://momjian.us EDB https://enterprisedb.com When a patient asks the doctor, "Am I going to die?", he means "Am I going to die soon?"
On Thu, Sep 26, 2024 at 03:08:52PM +0530, Amit Kapila wrote: > On Sat, Sep 21, 2024 at 1:50 AM Bruce Momjian <bruce@momjian.us> wrote: > > > > On Fri, Sep 20, 2024 at 04:05:11PM -0400, Tom Lane wrote: > > > Bruce Momjian <bruce@momjian.us> writes: > > > > Patch applied to PG 17. > > > > > > I don't see a push? > > > > Push was delayed because my test script found some uncommitted files due > > to earlier testing. Should be fine now. > > > > <para> > <link > linkend="app-pgcreatesubscriber"><application>pg_createsubscriber</application></link>, > a utility that creates logical replicas from physical standbys > </para> > > This description is okay but according to me, the more compelling use > case is that this new utility helps to allow online upgrades of > physical replication setup as explained in the blog [1]. See the > section: "Upgrading Streaming (Physical) Replication Setup". > > </listitem> > <listitem> > <para> > <link > linkend="pgupgrade"><application>pg_upgrade</application></link> now > preserves replication slots on both publishers and subscribers > </para> > > It is better to write the above statement as: > "pg_upgrade</application></link> now preserves replication slots on > publishers and full subscription's state on subscribers". This is > because replication slots are preserved on publishers. The subscribers > preserve the subscription state. So, as I understand it, this preservation only happens when the _old_ Postgres version is 17+. Do we want to try and explain that in the Postgres 17 release notes? -- Bruce Momjian <bruce@momjian.us> https://momjian.us EDB https://enterprisedb.com When a patient asks the doctor, "Am I going to die?", he means "Am I going to die soon?"
On Sun, Sep 29, 2024 at 6:50 AM Bruce Momjian <bruce@momjian.us> wrote: > > On Thu, Sep 26, 2024 at 03:08:52PM +0530, Amit Kapila wrote: > > On Sat, Sep 21, 2024 at 1:50 AM Bruce Momjian <bruce@momjian.us> wrote: > > > > > > On Fri, Sep 20, 2024 at 04:05:11PM -0400, Tom Lane wrote: > > > > Bruce Momjian <bruce@momjian.us> writes: > > > > > Patch applied to PG 17. > > > > > > > > I don't see a push? > > > > > > Push was delayed because my test script found some uncommitted files due > > > to earlier testing. Should be fine now. > > > > > > > <para> > > <link > > linkend="app-pgcreatesubscriber"><application>pg_createsubscriber</application></link>, > > a utility that creates logical replicas from physical standbys > > </para> > > > > This description is okay but according to me, the more compelling use > > case is that this new utility helps to allow online upgrades of > > physical replication setup as explained in the blog [1]. See the > > section: "Upgrading Streaming (Physical) Replication Setup". > > > > </listitem> > > <listitem> > > <para> > > <link > > linkend="pgupgrade"><application>pg_upgrade</application></link> now > > preserves replication slots on both publishers and subscribers > > </para> > > > > It is better to write the above statement as: > > "pg_upgrade</application></link> now preserves replication slots on > > publishers and full subscription's state on subscribers". This is > > because replication slots are preserved on publishers. The subscribers > > preserve the subscription state. > > So, as I understand it, this preservation only happens when the _old_ > Postgres version is 17+. > Yes. > Do we want to try and explain that in the > Postgres 17 release notes? > It would be good if we can capture that information without bloating the release document. However, this information is already present in pg_upgrade docs, so users have a way to know the same even if we can't mention it in the release notes. -- With Regards, Amit Kapila.
On Sat, 28 Sep 2024 21:19:11 -0400 Bruce Momjian <bruce@momjian.us> wrote: > On Thu, Sep 26, 2024 at 02:19:21PM +0900, Yugo Nagata wrote: > > On Thu, 9 May 2024 00:03:50 -0400 > > Bruce Momjian <bruce@momjian.us> wrote: > > > > > I have committed the first draft of the PG 17 release notes; you can > > > see the results here: > > > > I propose to improve the following description in "Migration to Version 17" > > section by adding CREATE INDEX and CREATE MATERIALIZED VIEW into the command list. > > > > <para> > > Change functions to use a safe <xref linkend="guc-search-path"/> > > during maintenance operations (Jeff Davis) > > <ulink url="&commit_baseurl;2af07e2f7">§</ulink> > > </para> > > > > It is suggested in the thread [1] that users could not notice the behaviour > > of CREATE INDEX is changed because the explicit command name is not listed in > > the release notes. So, I think it is better to add CREATE INDEX and > > CREATE MATERIALIZED VIEW into the command list. > > > > I've attached a patch. > > It this a valid change? Seems so. Yes. This change on CREATE INDEX was introduced by 2af07e2f7 together with other commands, but it was missed to be mentioned in the commit message although the description was added to the documentation. The change on CEATE MATERIALIZED VIEW was introduced by a separate commit b4da732fd, since which the REFRESH logic is used when creating a matview. Should we add here a link to that commit, too? Regards, Yugo Nagata > -- > Bruce Momjian <bruce@momjian.us> https://momjian.us > EDB https://enterprisedb.com > > When a patient asks the doctor, "Am I going to die?", he means > "Am I going to die soon?" -- Yugo NAGATA <nagata@sraoss.co.jp>
On Tue, Oct 1, 2024 at 04:36:09PM +0200, Laurenz Albe wrote: > I think that the removal of the "adminpack" extension should > be listed in the section "migration to v17" as an incompatibility. > > I have seen one complaint that pg_upgrade fails if the extension > is installed, but a dump/restore would also throw an error. Agreed. moved. -- Bruce Momjian <bruce@momjian.us> https://momjian.us EDB https://enterprisedb.com When a patient asks the doctor, "Am I going to die?", he means "Am I going to die soon?"
On Tue, Oct 8, 2024 at 6:25 AM Bruce Momjian <bruce@momjian.us> wrote: > > On Sun, Sep 29, 2024 at 06:33:29PM +0530, Amit Kapila wrote: > > > > It is better to write the above statement as: > > > > "pg_upgrade</application></link> now preserves replication slots on > > > > publishers and full subscription's state on subscribers". This is > > > > because replication slots are preserved on publishers. The subscribers > > > > preserve the subscription state. > > > > > > So, as I understand it, this preservation only happens when the _old_ > > > Postgres version is 17+. > > > > > > > Yes. > > > > > Do we want to try and explain that in the > > > Postgres 17 release notes? > > > > > > > It would be good if we can capture that information without bloating > > the release document. However, this information is already present in > > pg_upgrade docs, so users have a way to know the same even if we can't > > mention it in the release notes. > > I have developed the attached patch to mention it is "logical" slots, > and to mention its future use. > LGTM. Thanks! -- With Regards, Amit Kapila.
On Mon, 2024-10-07 at 20:11 -0400, Bruce Momjian wrote: > On Tue, Oct 1, 2024 at 04:36:09PM +0200, Laurenz Albe wrote: > > I think that the removal of the "adminpack" extension should > > be listed in the section "migration to v17" as an incompatibility. > > > > I have seen one complaint that pg_upgrade fails if the extension > > is installed, but a dump/restore would also throw an error. > > Agreed. moved. Thank you! Yours, Laurenz Albe
On Mon, 7 Oct 2024 20:00:45 -0400 Bruce Momjian <bruce@momjian.us> wrote: > On Mon, Oct 7, 2024 at 07:25:11PM -0400, Bruce Momjian wrote: > > > Yes. This change on CREATE INDEX was introduced by 2af07e2f7 together with > > > other commands, but it was missed to be mentioned in the commit message > > > although the description was added to the documentation. > > > > > > The change on CEATE MATERIALIZED VIEW was introduced by a separate commit > > > b4da732fd, since which the REFRESH logic is used when creating a matview. > > > Should we add here a link to that commit, too? > > > > I developed the attached patch which adds the two commands and the > > commit item. > > Okay, updated commit after running src/tools/add_commit_links.pl. LGTM. Thank you. Regards, Yugo Nagata -- Yugo NAGATA <nagata@sraoss.co.jp>
On Tue, Oct 8, 2024 at 07:51:23PM +0900, Yugo NAGATA wrote: > On Mon, 7 Oct 2024 20:00:45 -0400 > Bruce Momjian <bruce@momjian.us> wrote: > > > On Mon, Oct 7, 2024 at 07:25:11PM -0400, Bruce Momjian wrote: > > > > Yes. This change on CREATE INDEX was introduced by 2af07e2f7 together with > > > > other commands, but it was missed to be mentioned in the commit message > > > > although the description was added to the documentation. > > > > > > > > The change on CEATE MATERIALIZED VIEW was introduced by a separate commit > > > > b4da732fd, since which the REFRESH logic is used when creating a matview. > > > > Should we add here a link to that commit, too? > > > > > > I developed the attached patch which adds the two commands and the > > > commit item. > > > > Okay, updated commit after running src/tools/add_commit_links.pl. > > LGTM. Thank you. Thanks, patch applied. -- Bruce Momjian <bruce@momjian.us> https://momjian.us EDB https://enterprisedb.com When a patient asks the doctor, "Am I going to die?", he means "Am I going to die soon?"
On Tue, Oct 8, 2024 at 11:57:19AM +0530, Amit Kapila wrote: > On Tue, Oct 8, 2024 at 6:25 AM Bruce Momjian <bruce@momjian.us> wrote: > > > > On Sun, Sep 29, 2024 at 06:33:29PM +0530, Amit Kapila wrote: > > > > > It is better to write the above statement as: > > > > > "pg_upgrade</application></link> now preserves replication slots on > > > > > publishers and full subscription's state on subscribers". This is > > > > > because replication slots are preserved on publishers. The subscribers > > > > > preserve the subscription state. > > > > > > > > So, as I understand it, this preservation only happens when the _old_ > > > > Postgres version is 17+. > > > > > > > > > > Yes. > > > > > > > Do we want to try and explain that in the > > > > Postgres 17 release notes? > > > > > > > > > > It would be good if we can capture that information without bloating > > > the release document. However, this information is already present in > > > pg_upgrade docs, so users have a way to know the same even if we can't > > > mention it in the release notes. > > > > I have developed the attached patch to mention it is "logical" slots, > > and to mention its future use. > > > > LGTM. Thanks! Great, patch applied. -- Bruce Momjian <bruce@momjian.us> https://momjian.us EDB https://enterprisedb.com When a patient asks the doctor, "Am I going to die?", he means "Am I going to die soon?"