== PostgreSQL Weekly News - March 22, 2020 == - Mailing list pgsql-announce
| From | David Fetter |
|---|---|
| Subject | == PostgreSQL Weekly News - March 22, 2020 == |
| Date | |
| Msg-id | 20200322225119.GA1969@fetter.org Whole thread Raw |
| List | pgsql-announce |
== PostgreSQL Weekly News - March 22, 2020 == The German-speaking PostgreSQL conference and Swiss PGDay have been cancelled due to COVID-19. == PostgreSQL Product News == pgBadger v11.2, a PostgreSQL log analyzer and graph tool written in Perl, released. https://github.com/darold/pgbadger/releases check_pgbackrest 1.8, a Nagios-compatible monitor for pgBackRest, released. https://github.com/dalibo/check_pgbackrest/releases == PostgreSQL Jobs for March == http://archives.postgresql.org/pgsql-jobs/2020-03/ == PostgreSQL Local == PGDay Ukraine will take place April 25th, 2020 in Lviv at the Bank Hotel. https://pgday.org.ua/ PgConf Belgium will take place on May 8, 2020 at the UCLL Campus in Haasrode (near Leuven). The CfS is open through March 31, 2020 at https://pgconf.be/#cfs http://pgconf.be PGCon 2020 will take place in Ottawa on May 26-29, 2020. https://www.pgcon.org/2020/ PGDay.IT 2020 will take place June 11-12 in Bergamo, Italy. https://2020.pgday.it/en/ Postgres Ibiza will be held in Ibiza, Spain on June 25-26, 2020. https://pgibz.io/ PostgresLondon 2020 will be July 7-8, 2020 with an optional training day on July 6. The CfP is open at https://forms.gle/5m8ybUt9YDZG4gVU7 through March 27, 2020. http://postgreslondon.org PG Day Russia will be in Saint Petersburg on July 10, 2020. The CfP is open at https://pgday.ru/en/2020/for-speakers through April 6, 2020. https://pgday.ru/en/2020/ FOSS4G 2020, will take place in Calgary, Alberta, Canada August 24-29 2020. the Call for Papers is currently open at https://2020.foss4g.org/speakers/ https://2020.foss4g.org/ pgDay Israel 2020 will take place on September 10, 2020 in Tel Aviv. http://pgday.org.il/ Austrian pgDay will take place September 18, 2020 at Schloss Schoenbrunn (Apothekertrakt) in Vienna. The CfP is open until April 19, 2020 at https://pgday.at/en/talk-commitee/ https://pgday.at/en/ == PostgreSQL in the News == Planet PostgreSQL: http://planet.postgresql.org/ PostgreSQL Weekly News is brought to you this week by David Fetter Submit news and announcements by Sunday at 3:00pm PST8PDT to david@fetter.org. == Applied Patches == Peter Geoghegan pushed: - nbtree: Reorder nbtinsert.c prototypes. Relocate _bt_newroot() prototype, so that the order that prototypes appear in matches the order that the functions are defined in. https://git.postgresql.org/pg/commitdiff/f207bb0b8f13999c91b405a2e6c8526225470816 - nbtree: Pass down MAXALIGN()'d itemsz for new item. Refactor nbtinsert.c so that the final itemsz of each new non-pivot tuple (the MAXALIGN()'d size) is determined once. Most of the functions used by leaf page inserts used the insertstate.itemsz value already. This commit makes everything use insertstate.itemsz as standard practice. The goal is to decouple tuple size from "effective" tuple size. Making this distinction isn't truly necessary right now, but that might change in the future. Also explain why we consistently apply MAXALIGN() to get an effective index tuple size. This was rather unclear, in part because it isn't actually strictly necessary right now. https://git.postgresql.org/pg/commitdiff/013c1f6af6c5017d97b1268cf9a5fa3d0575eecb - nbtree: Fix obsolete _bt_search() comment. Oversight in commit d2086b08b02. https://git.postgresql.org/pg/commitdiff/113758155c11cf993ca0ecee8856e300a2525a30 - Doc: Correct deduplicate_items varlistentry id. Use a varlistentry id for the deduplicate_items storage parameter that is derived from the name of the parameter itself. This oversight happened because the storage parameter was renamed relatively late during the development of the patch that became commit 0d861bbb. https://git.postgresql.org/pg/commitdiff/dbbb55385cf5eb75837eb3eb7ca1f14e2beb5a63 - nbtree: Remove useless local variables. Copying block and offset numbers to local variables in _bt_insertonpg() made the code less readable. Remove the variables. There is already code that conditionally calls BufferGetBlockNumber() in the same block, so consistently do it that way instead. Calling BufferGetBlockNumber() is very cheap, but we might as well avoid it when it isn't truly necessary. It isn't truly necessary for _bt_insertonpg() to call BufferGetBlockNumber() in almost all cases. Spotted while working on a patch that refactors the fastpath rightmost leaf page cache optimization, which was added by commit 2b272734. https://git.postgresql.org/pg/commitdiff/b897b3aae6b525922fe3d074d4bdf5f2674954dd - Refactor nbtree fastpath optimization. Commit 2b272734, which added the fastpath rightmost leaf page cache insert optimization, added code to _bt_doinsert() to handle using and invalidating the backend local block cache. It doesn't seem like a good place to handle these low level details, though. _bt_doinsert() is supposed to be a high level function -- it is the main entry point to nbtinsert.c. Restructure the code by placing handling of the rightmost block cache at the start of a new _bt_search() shim function, _bt_search_insert(). The new function is called from _bt_doinsert(), which uses it as a _bt_search() variant that conveniently accepts its BTInsertState state as an argument. _bt_doinsert() no longer needs to directly consider the fastpath optimization. Discussion: https://postgr.es/m/CAH2-Wzk59cxKJRd=rfbyub6-V4yWRjsOYRkUNHBLT1P1GdtCQQ@mail.gmail.com https://git.postgresql.org/pg/commitdiff/b029395f5e616e0f2b1131b2c7ecb6640f30c055 - nbtree: Use raw PageAddItem() for retail inserts. Only internal page splits need to call _bt_pgaddtup() instead of PageAddItem(), and only for data items, one of which will end up at the first offset (or first offset after the high key offset) on the new right page. This data item alone will need to be truncated in _bt_pgaddtup(). Since there is no reason why retail inserts ever need to truncate the incoming item, use a raw PageAddItem() call there instead. Even _bt_split() uses raw PageAddItem() calls for left page and right page high keys. Clearly the _bt_pgaddtup() shim function wasn't really encapsulating anything. _bt_pgaddtup() should now be thought of as a _bt_split() helper function. Note that the assertions from commit d1e241c2 verify that retail inserts never insert an item at an internal page's negative infinity offset. This invariant could only ever be violated as a result of a basic logic error in nbtinsert.c. https://git.postgresql.org/pg/commitdiff/6312c08a291fc34068571ec64ee439f2674d739a - nbtree: Remove obsolete _bt_pgaddtup() comments. Remove comments that are a throw back to a time when nbtree cared about write-ordering dependencies. The comments are similar to those removed by commit 9ee7414e, among others. https://git.postgresql.org/pg/commitdiff/b27e1b3418b189e1a5aee13cbd09bff94f010a86 Thomas Munro pushed: - Simplify the effective_io_concurrency setting. The effective_io_concurrency GUC and equivalent tablespace option were previously passed through a formula based on a theory about RAID spindles and probabilities, to arrive at the number of pages to prefetch in bitmap heap scans. Tomas Vondra, Andres Freund and others argued that it was anachronistic and hard to justify, and commit 558a9165e08 already started down the path of bypassing it in new code. We agreed to drop that logic and use the value directly. For the default setting of 1, there is no change in effect. Higher settings can be converted from the old meaning to the new with: select round(sum(OLD / n::float)) from generate_series(1, OLD) s(n); We might want to consider renaming the GUC before the next release given the change in meaning, but it's not clear that many users had set it very carefully anyway. That decision is deferred for now. Discussion: https://postgr.es/m/CA%2BhUKGJUw08dPs_3EUcdO6M90GnjofPYrWp4YSLaBkgYwS-AqA%40mail.gmail.com https://git.postgresql.org/pg/commitdiff/b09ff53667ffc986371ec8ffa372916ad460220d - Introduce a maintenance_io_concurrency setting. Introduce a GUC and a tablespace option to control I/O prefetching, much like effective_io_concurrency, but for work that is done on behalf of many client sessions. Use the new setting in heapam.c instead of the hard-coded formula effective_io_concurrency + 10 introduced by commit 558a9165e08. Go with a default value of 10 for now, because it's a round number pretty close to the value used for that existing case. Discussion: https://postgr.es/m/CA%2BhUKGJUw08dPs_3EUcdO6M90GnjofPYrWp4YSLaBkgYwS-AqA%40mail.gmail.com https://git.postgresql.org/pg/commitdiff/fc34b0d9de27ab5ac2887da00b3892bdabd89e45 - Fix kqueue support under debugger on macOS. While running under a debugger, macOS's getppid() can return the debugger's PID. That could cause a backend to exit because it falsely believed that the postmaster had died, since commit 815c2f09. Continue to use getppid() as a fast postmaster check after adding the postmaster's PID to a kqueue, to close a PID-reuse race, but double check that it actually exited by trying to read the pipe. The new check isn't reached in the common case. Reported-by: Alexander Korotkov <a.korotkov@postgrespro.ru> Discussion: https://postgr.es/m/CA%2BhUKGKhAxJ8V8RVwCo6zJaeVrdOG1kFBHGZOOjf6DzW_omeMA%40mail.gmail.com https://git.postgresql.org/pg/commitdiff/7bc84a1f304acc9f4705c4178eb362ddce310452 - Don't use EV_CLEAR for kqueue events. For the semantics to match the epoll implementation, we need a socket to continue to appear readable/writable if you wait multiple times without doing I/O in between (in Linux terminology: level-triggered rather than edge-triggered). This distinction will be important for later commits. Similar to commit 3b790d256f8 for Windows. Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://postgr.es/m/CA%2BhUKGJAC4Oqao%3DqforhNey20J8CiG2R%3DoBPqvfR0vOJrFysGw%40mail.gmail.com https://git.postgresql.org/pg/commitdiff/9b8aa0929390aab1f2f0859016e6110d283732dd Álvaro Herrera pushed: - Plug memory leak. Introduced by b08dee24a557. Noted by Coverity. https://git.postgresql.org/pg/commitdiff/046001fe3cf59e19b2e8c24b82fb2e9b8ab6edd4 - Update comment. I forgot to update it per last-minute edits leading to commit 2f9661311b83. Reported by Mark Dilger. Discussion: https://postgr.es/m/DA9A5554-AADD-4F33-96F9-A1066EC9CAB5@enterprisedb.com> https://git.postgresql.org/pg/commitdiff/b234b567ed740aae368672670d8d03640e884f1f - Document pg_ls_*dir hiding of directories and special files. It's strange that a directory-listing function does not list all entries in a directory, so let's at least document it. This involves pg_ls_logdir pg_ls_waldir pg_ls_archive_statusdir pg_ls_tmpdir Backpatch as far back as it applies cleanly (and as far as as each function exists). REL_10_STABLE uses different wording, but hopefully people are not reading docs so old to write new apps anyway. Author: Justin Pryzby <pryzby@telsasoft.com> Discussion: https://postgr.es/m/20200305161838.GJ684@telsasoft.com https://git.postgresql.org/pg/commitdiff/8b6d94cf6c8319bfd6bebf8b863a5db586c19c3b - Fix consistency issues with replication slot copy. Commit 9f06d79ef831's replication slot copying failed to properly reserve the WAL that the slot is expecting to see during DecodingContextFindStartpoint (to set the confirmed_flush LSN), so concurrent activity could remove that WAL and cause the copy process to error out. But it doesn't actually *need* that WAL anyway: instead of running decode to find confirmed_flush, it can be copied from the source slot. Fix this by rearranging things to avoid DecodingContextFindStartpoint() (leaving the target slot's confirmed_flush_lsn to invalid), and set that up afterwards by copying from the target slot's value. Also ensure the source slot's confirmed_flush_lsn is valid. Reported-by: Arseny Sher Author: Masahiko Sawada, Arseny Sher Discussion: https://postgr.es/m/871rr3ohbo.fsf@ars-thinkpad https://git.postgresql.org/pg/commitdiff/bcd1c3630095e48bc3b1eb0fc8e8c8a7c851eba1 - Remove logical_read_local_xlog_page. It devolved into a content-less wrapper over read_local_xlog_page, with nothing to add, plus it's easily confused with walsender's logical_read_xlog_page. There doesn't seem to be any reason for it to stay. src/include/replication/logicalfuncs.h becomes empty, so remove it too. The prototypes it initially had were absorbed by generated fmgrprotos.h. Discussion: https://postgr.es/m/20191115214102.GA15616@alvherre.pgsql https://git.postgresql.org/pg/commitdiff/5d0c2d5eba6b58b3c9955ee41c917f7df8b62746 - Enable BEFORE row-level triggers for partitioned tables. ... with the limitation that the tuple must remain in the same partition. Reviewed-by: Ashutosh Bapat Discussion: https://postgr.es/m/20200227165158.GA2071@alvherre.pgsql https://git.postgresql.org/pg/commitdiff/487e9861d0cf83e9100ad0d0369147db3ef4ea73 - pg_dump: Add FOREIGN to ALTER statements, if appropriate. Author: Luis Carril Reviewed-by: Tomas Vondra, Daniel Gustafsson, Álvaro Herrera Discussion: https://postgr.es/m/LEJPR01MB0185A19B2E7C98E5E2A031F5E7F20@LEJPR01MB0185.DEUPRD01.PROD.OUTLOOK.DE https://git.postgresql.org/pg/commitdiff/4e62091341b4a55666a32dd7762f185a1faa54f7 - Fix bogus last-minute edit in 4e62091341b4. Noticed by Erik Rijkers before I was able to push the fix. https://git.postgresql.org/pg/commitdiff/069b750ca764a85324a138c22c3ece5cf13d684f Tom Lane pushed: - Avoid holding a directory FD open across assorted SRF calls. This extends the fixes made in commit 085b6b667 to other SRFs with the same bug, namely pg_logdir_ls(), pgrowlocks(), pg_timezone_names(), pg_ls_dir(), and pg_tablespace_databases(). Also adjust various comments and documentation to warn against expecting to clean up resources during a ValuePerCall SRF's final call. Back-patch to all supported branches, since these functions were all born broken. Justin Pryzby, with cosmetic tweaks by me Discussion: https://postgr.es/m/20200308173103.GC1357@telsasoft.com https://git.postgresql.org/pg/commitdiff/b4570d33aa045df330bb325ba8a2cbf02266a555 - Remove useless pfree()s at the ends of various ValuePerCall SRFs. We don't need to manually clean up allocations in a SRF's multi_call_memory_ctx, because the SRF_RETURN_DONE infrastructure takes care of that (and also ensures that it will happen even if the function never gets a final call, which simple manual cleanup cannot do). Hence, the code removed by this patch is a waste of code and cycles. Worse, it gives the impression that cleaning up manually is a thing, which can lead to more serious errors such as those fixed in commits 085b6b667 and b4570d33a. So we should get rid of it. These are not quite actual bugs though, so I couldn't muster the enthusiasm to back-patch. Fix in HEAD only. Justin Pryzby Discussion: https://postgr.es/m/20200308173103.GC1357@telsasoft.com https://git.postgresql.org/pg/commitdiff/41b45576d532ab6ef4a968f78b5b06d700ebf61f - Use pkg-config, if available, to locate libxml2 during configure. If pkg-config is installed and knows about libxml2, use its information rather than asking xml2-config. Otherwise proceed as before. This patch allows "configure --with-libxml" to succeed on platforms that have pkg-config but not xml2-config, which is likely to soon become a typical situation. The old mechanism can be forced by setting XML2_CONFIG explicitly (hence, build processes that were already doing so will certainly not need adjustment). Also, it's now possible to set XML2_CFLAGS and XML2_LIBS explicitly to override both programs. There is a small risk of this breaking existing build processes, if there are multiple libxml2 installations on the machine and pkg-config disagrees with xml2-config about which to use. The only case where that seems really likely is if a builder has tried to select a non-default xml2-config by putting it early in his PATH rather than setting XML2_CONFIG. Plan to warn against that in the minor release notes. Back-patch to v10; before that we had no pkg-config infrastructure, and it doesn't seem worth adding it for this. Hugh McMaster and Tom Lane; Peter Eisentraut also made an earlier attempt at this, from which I lifted most of the docs changes. Discussion: https://postgr.es/m/CAN9BcdvfUwc9Yx5015bLH2TOiQ-M+t_NADBSPhMF7dZ=pLa_iw@mail.gmail.com https://git.postgresql.org/pg/commitdiff/0bc8cebdb889368abdf224aeac8bc197fe4c9ae6 - Remove bogus assertion about polymorphic SQL function result. It is possible to reach check_sql_fn_retval() with an unresolved polymorphic rettype, resulting in an assertion failure as demonstrated by one of the added test cases. However, the code following that throws what seems an acceptable error message, so just remove the Assert and adjust commentary. While here, I thought it'd be a good idea to provide some parallel tests of SQL-function and PL/pgSQL-function polymorphism behavior. Some of these cases are perhaps duplicative of tests elsewhere, but we hadn't any organized coverage of the topic AFAICS. Although that assertion's been wrong all along, it won't have any effect in production builds, so I'm not bothering to back-patch. Discussion: https://postgr.es/m/21569.1584314271@sss.pgh.pa.us https://git.postgresql.org/pg/commitdiff/9d9784c840f3ac98eb41d021c981eea0fe1735ed - Doc: clarify behavior of "anyrange" pseudo-type. I noticed that we completely failed to document the restriction that an "anyrange" result type has to be inferred from an "anyrange" input. The docs also were less clear than they could be about the relationship between "anyrange" and "anyarray". It's been like this all along, so back-patch. https://git.postgresql.org/pg/commitdiff/31d846e0265c2c1415d7910d39d5b259b92184ea - Adjust handling of an ANYARRAY actual input for an ANYARRAY argument. Ordinarily it's impossible for an actual input of a function to have declared type ANYARRAY, since we'd resolve that to a concrete array type before doing argument type resolution for the function. But an exception arises for functions applied to certain columns of pg_statistic or pg_stats, since we abuse the "anyarray" pseudotype by using it to declare those columns. So parse_coerce.c has to deal with the case. Previously we allowed an ANYARRAY actual input to match an ANYARRAY polymorphic argument, but only if no other argument or result was declared ANYELEMENT. When that logic was written, those were the only two polymorphic types, and I fear nobody thought carefully about how it ought to extend to the other ones that came along later. But actually it was wrong even then, because if a function has two ANYARRAY arguments, it should be able to expect that they have identical element types, and we'd not be able to ensure that. The correct generalization is that we can match an ANYARRAY actual input to an ANYARRAY polymorphic argument only if no other argument or result is of any polymorphic type, so that no promises are being made about element type compatibility. check_generic_type_consistency can't test that condition, but it seems better anyway to accept such matches there and then throw an error if needed in enforce_generic_type_consistency. That way we can produce a specific error message rather than an unintuitive "function does not exist" complaint. (There's some risk perhaps of getting new ambiguous-function complaints, but I think that any set of functions for which that could happen would be ambiguous against ordinary array columns as well.) While we're at it, we can improve the message that's produced in cases that the code did already object to, as shown in the regression test changes. Also, remove a similar test that got cargo-culted in for ANYRANGE; there are no catalog columns of type ANYRANGE, and I hope we never create any, so that's not needed. (It was incomplete anyway.) While here, update some comments and rearrange the code a bit in preparation for upcoming additions of more polymorphic types. In practical situations I believe this is just exchanging one error message for another, hopefully better, one. So it doesn't seem needful to back-patch, even though the mistake is ancient. Discussion: https://postgr.es/m/21569.1584314271@sss.pgh.pa.us https://git.postgresql.org/pg/commitdiff/77ec5affbc44cc0a1d06d4201f0d7c8c79371550 - Refactor our checks for valid function and aggregate signatures. pg_proc.c and pg_aggregate.c had near-duplicate copies of the logic to decide whether a function or aggregate's signature is legal. This seems like a bad thing even without the problem that the upcoming "anycompatible" patch would roughly double the complexity of that logic. Hence, refactor so that the rules are localized in new subroutines supplied by parse_coerce.c. (One could quibble about just where to add that code, but putting it beside enforce_generic_type_consistency seems not totally unreasonable.) The fact that the rules are about to change would mandate some changes in the wording of the associated error messages in any case. I ended up spelling things out in a fairly literal fashion in the errdetail messages, eg "A result of type anyelement requires at least one input of type anyelement, anyarray, anynonarray, anyenum, or anyrange." Perhaps this is overkill, but once there's more than one subgroup of polymorphic types, people might get confused by more-abstract messages. Discussion: https://postgr.es/m/24137.1584139352@sss.pgh.pa.us https://git.postgresql.org/pg/commitdiff/e6c178b5b73ac1fb822829e6d9b99e3fc1129c21 - Introduce "anycompatible" family of polymorphic types. This patch adds the pseudo-types anycompatible, anycompatiblearray, anycompatiblenonarray, and anycompatiblerange. They work much like anyelement, anyarray, anynonarray, and anyrange respectively, except that the actual input values need not match precisely in type. Instead, if we can find a common supertype (using the same rules as for UNION/CASE type resolution), then the parser automatically promotes the input values to that type. For example, "myfunc(anycompatible, anycompatible)" can match a call with one integer and one bigint argument, with the integer automatically promoted to bigint. With anyelement in the definition, the user would have had to cast the integer explicitly. The new types also provide a second, independent set of type variables for function matching; thus with "myfunc(anyelement, anyelement, anycompatible) returns anycompatible" the first two arguments are constrained to be the same type, but the third can be some other type, and the result has the type of the third argument. The need for more than one set of type variables was foreseen back when we first invented the polymorphic types, but we never did anything about it. Pavel Stehule, revised a bit by me Discussion: https://postgr.es/m/CAFj8pRDna7VqNi8gR+Tt2Ktmz0cq5G93guc3Sbn_NVPLdXAkqA@mail.gmail.com https://git.postgresql.org/pg/commitdiff/24e2885ee304cb6a94fdfc25a1a108344ed9f4f7 - Fix up recent breakage of headerscheck and cpluspluscheck. headerscheck and cpluspluscheck should skip the recently-added cmdtaglist.h header, since (like kwlist.h and some other similarly- designed headers) it's not meant to be included standalone. evtcache.h was missing an #include to support its usage of Bitmapset. typecmds.h was missing an #include to support its usage of ParseState. The first two of these were evidently oversights in commit 2f9661311. I didn't track down exactly which change broke typecmds.h, but it must have been some rearrangement in one of its existing inclusions, because it's referenced ParseState for quite a long time and there were not complaints from these checking programs before. https://git.postgresql.org/pg/commitdiff/d0587f52b3bb898db3c0011954de6ae9adc076c8 Fujii Masao pushed: - Fix comment in xlog.c. This commit fixes the comment about SharedHotStandbyActive variable. The comment was apparently copy-and-pasted. Author: Atsushi Torikoshi Discussion: https://postgr.es/m/CACZ0uYEjpqZB9wN2Rwc_RMvDybyYqdbkPuDr1NyxJg4f9yGfMw@mail.gmail.com https://git.postgresql.org/pg/commitdiff/1429d3f767a05e7a4819f55c5b3ff768862dcf78 - Add the type information for index storage parameters to the documentation. Author: Atsushi Torikoshi Discussion: https://postgr.es/m/CACZ0uYFQebs4WT5eu3dK4qm_2PurZuvB++8nDvSBG0ebRWmbdg@mail.gmail.com https://git.postgresql.org/pg/commitdiff/28e0a103a835aec8b31c897b19b3ceb7d92e7c12 - Update the description of type of check_option reloption in docs. Commit 773df883e8f7 changed the type of check_option reloption from string to enum. But it forgot to update the description of the type in the documentation. Author: Atsushi Torikoshi Discussion: https://postgr.es/m/CACZ0uYFvHF4n6yxF390YZgr4Q0Z0c2w0ihu=DLb8ipNOnNcqzQ@mail.gmail.com https://git.postgresql.org/pg/commitdiff/1558413432e3dce941af550f1ad0d41790ec6fba - Correct the descriptions of recovery-related wait events in docs. This commit corrects the descriptions of RecoveryWalAll and RecoveryWalStream wait events in the documentation. Back-patch to v10 where those wait events were added. Author: Fujii Masao Reviewed-by: Kyotaro Horiguchi, Atsushi Torikoshi Discussion: https://postgr.es/m/124997ee-096a-5d09-d8da-2c7a57d0816e@oss.nttdata.com https://git.postgresql.org/pg/commitdiff/dbe0d9892c338a475c0e791a5e77cff798216d29 - Rename the recovery-related wait events. This commit renames RecoveryWalAll and RecoveryWalStream wait events to RecoveryWalStream and RecoveryRetrieveRetryInterval, respectively, in order to make the names and what they are more consistent. For example, previously RecoveryWalAll was reported as a wait event while the recovery was waiting for WAL from a stream, and which was confusing because the name was very different from the situation where the wait actually could happen. The names of macro variables for those wait events also are renamed accordingly. This commit also changes the category of RecoveryRetrieveRetryInterval to Timeout from Activity because the wait event is reported while waiting based on wal_retrieve_retry_interval. Author: Fujii Masao Reviewed-by: Kyotaro Horiguchi, Atsushi Torikoshi Discussion: https://postgr.es/m/124997ee-096a-5d09-d8da-2c7a57d0816e@oss.nttdata.com https://git.postgresql.org/pg/commitdiff/1d253bae57fcb3e75cdd9cdb2592c5b09cf7e7b7 - Make pg_basebackup ask the server to estimate the total backup size, by default. This commit changes pg_basebackup so that it specifies PROGRESS option in BASE_BACKUP replication command whether --progress is specified or not. This causes the server to estimate the total backup size and report it in pg_stat_progress_basebackup.backup_total, by default. This is reasonable default because the time required for the estimation would not be so large in most cases. Also this commit adds new option --no-estimate-size to pg_basebackup. This option prevents the server from the estimation, and so is useful to avoid such estimation time if it's too long. Author: Fujii Masao Reviewed-by: Magnus Hagander, Amit Langote Discussion: https://postgr.es/m/CABUevEyDPPSjP7KRvfTXPdqOdY5aWNkqsB5aAXs3bco5ZwtGHg@mail.gmail.com https://git.postgresql.org/pg/commitdiff/fab13dc50ba5e7a12b474a7366024681bc169ac8 Amit Kapila pushed: - Assert that we don't acquire a heavyweight lock on another object after.relation extension lock. The only exception to the rule is that we can try to acquire the same relation extension lock more than once. This is allowed as we are not creating any new lock for this case. This restriction implies that the relation extension lock won't ever participate in the deadlock cycle because we can never wait for any other heavyweight lock after acquiring this lock. Such a restriction is okay for relation extension locks as unlike other heavyweight locks these are not held till the transaction end. These are taken for a short duration to extend a particular relation and then released. Author: Dilip Kumar, with few changes by Amit Kapila Reviewed-by: Amit Kapila, Kuntal Ghosh and Sawada Masahiko Discussion: https://postgr.es/m/CAD21AoCmT3cFQUN4aVvzy5chw7DuzXrJCbrjTU05B+Ss=Gn1LA@mail.gmail.com https://git.postgresql.org/pg/commitdiff/15ef6ff4b985276c386adf3e11ebf7f955ea6f1f - Add missing errcode() in a few ereport calls. This will allow to specifying SQLSTATE error code for the errors in the missing places. Reported-by: Sawada Masahiko Author: Sawada Masahiko Backpatch-through: 9.5 Discussion: https://postgr.es/m/CA+fd4k6N8EjNvZpM8nme+y+05mz-SM8Z_BgkixzkA34R+ej0Kw@mail.gmail.com https://git.postgresql.org/pg/commitdiff/b4f140869f5da081c27c344587b0ff1a02821776 - Add assert to ensure that page locks don't participate in deadlock cycle. Assert that we don't acquire any other heavyweight lock while holding the page lock except for relation extension. However, these locks are never taken in reverse order which implies that page locks will never participate in the deadlock cycle. Similar to relation extension, page locks are also held for a short duration, so imposing such a restriction won't hurt. Author: Dilip Kumar, with few changes by Amit Kapila Reviewed-by: Amit Kapila, Kuntal Ghosh and Sawada Masahiko Discussion: https://postgr.es/m/CAD21AoCmT3cFQUN4aVvzy5chw7DuzXrJCbrjTU05B+Ss=Gn1LA@mail.gmail.com https://git.postgresql.org/pg/commitdiff/72e78d831ab5550c39f2dcc7cc5d44c406ec3dc2 - Allow relation extension lock to conflict among parallel group members. This is required as it is no safer for two related processes to extend the same relation at a time than for unrelated processes to do the same. We don't acquire a heavyweight lock on any other object after relation extension lock which means such a lock can never participate in the deadlock cycle. So, avoid checking wait edges from this lock. This provides an infrastructure to allow parallel operations like insert, copy, etc. which were earlier not possible as parallel group members won't conflict for relation extension lock. Author: Dilip Kumar, Amit Kapila Reviewed-by: Amit Kapila, Kuntal Ghosh and Sawada Masahiko Discussion: https://postgr.es/m/CAD21AoCmT3cFQUN4aVvzy5chw7DuzXrJCbrjTU05B+Ss=Gn1LA@mail.gmail.com https://git.postgresql.org/pg/commitdiff/85f6b49c2c53fb1e08d918ec9305faac13cf7ad6 - Allow page lock to conflict among parallel group members. This is required as it is no safer for two related processes to perform clean up in gin indexes at a time than for unrelated processes to do the same. After acquiring page locks, we can acquire relation extension lock but reverse never happens which means these will also not participate in deadlock. So, avoid checking wait edges from this lock. Currently, the parallel mode is strictly read-only, but after this patch we have the infrastructure to allow parallel inserts and parallel copy. Author: Dilip Kumar, Amit Kapila Reviewed-by: Amit Kapila, Kuntal Ghosh and Sawada Masahiko Discussion: https://postgr.es/m/CAD21AoCmT3cFQUN4aVvzy5chw7DuzXrJCbrjTU05B+Ss=Gn1LA@mail.gmail.com https://git.postgresql.org/pg/commitdiff/3ba59ccc896e8877e2fbfb8d4f148904cad5f9b0 Michaël Paquier pushed: - Fix typo in indexcmds.c. Introduced by 61d7c7b. Backpatch-through: 12 https://git.postgresql.org/pg/commitdiff/fdeeb524b4e28affc527548af389c75e0a99c340 - Fix comment related to concurrent index swapping in index.c. A comment about switching indisvalid of the new and old indexes swapped in REINDEX CONCURRENTLY got this backwards. Issue introduced by 5dc92b8, the original commit of REINDEX CONCURRENTLY. Author: Julien Rouhaud Discussion: https://postgr.es/m/20200318143340.GA46897@nol Backpatch-through: 12 https://git.postgresql.org/pg/commitdiff/d41202f36eed97a4ee68a6a9f5fcca314f12c35d Tomáš Vondra pushed: - Fix wording of several extended stats comments. Reported-by: Thomas Munro Discussion: https://www.postgresql.org/message-id/flat/20200113230008.g67iyk4cs3xbnjju@development https://git.postgresql.org/pg/commitdiff/6f72dbc48bf8a0f4dc3248691355b0688d7aeba2 - Recognize some OR clauses as compatible with functional dependencies. Since commit 8f321bd16c functional dependencies can handle IN clauses, which however introduced a possible (and surprising) inconsistency, because IN clauses may be expressed as an OR clause, which are still considered incompatible. For example a IN (1, 2, 3) may be rewritten as (a = 1 OR a = 2 OR a = 3) The IN clause will work fine with functional dependencies, but the OR clause will force the estimation to fall back to plain per-column estimates, possibly introducing significant estimation errors. This commit recognizes OR clauses equivalent to an IN clause (when all arugments are compatible and reference the same attribute) as a special case, compatible with functional dependencies. This allows applying functional dependencies, just like for IN clauses. This does not eliminate the difference in estimating the clause itself, i.e. IN clause and OR clause still use different formulas. It would be possible to change that (for these special OR clauses), but that's not really about extended statistics - it was always like this. Moreover the errors are usually much smaller compared to ignoring dependencies. Author: Tomas Vondra Reviewed-by: Dean Rasheed Discussion: https://www.postgresql.org/message-id/flat/13902317.Eha0YfKkKy%40pierred-pdoc https://git.postgresql.org/pg/commitdiff/ccaa3569f58796868303629bc2d63ddddb599b38 - Document pg_statistic_ext.stxstattarget. Commit d06215d03b added a new attribute to pg_statistic_ext catalog, but failed to add it to document it properly. Reported-by: Noriyoshi Shinoda <noriyoshi.shinoda@hpe.com> https://git.postgresql.org/pg/commitdiff/c31132d87c6315bbbe4b4aa383705aaae2348c0e Peter Eisentraut pushed: - doc: Update documentation about reg* types. Add missing index entries, add missing information on pg_upgrade man page, order things alphabetical instead of (apparently) in the order they were implemented, reduce repetitiveness a bit. https://git.postgresql.org/pg/commitdiff/8408e3a557ad26a7e88f867a425b2b9a86c4fa04 - Implement type regcollation. This will be helpful for a following commit and it's also just generally useful, like the other reg* types. Author: Julien Rouhaud Reviewed-by: Thomas Munro and Michael Paquier Discussion: https://postgr.es/m/CAEepm%3D0uEQCpfq_%2BLYFBdArCe4Ot98t1aR4eYiYTe%3DyavQygiQ%40mail.gmail.com https://git.postgresql.org/pg/commitdiff/a2b1faa0f279f18517adc1dd969e5f026eb51d33 - Prepare to support non-tables in publications. This by itself doesn't change any functionality but prepares the way for having relations other than base tables in publications. Make arrangements for COPY handling the initial table sync. For non-tables we have to use COPY (SELECT ...) instead of directly copying from the table, but then we have to take care to omit generated columns from the column list. Also, remove a hardcoded reference to relkind = 'r' and rely on the publisher to send only what it can actually publish, which will be correct even in future cross-version scenarios. Reviewed-by: Amit Langote <amitlangote09@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CA+HiwqH=Y85vRK3mOdjEkqFK+E=ST=eQiHdpj43L=_eJMOOznQ@mail.gmail.com https://git.postgresql.org/pg/commitdiff/c314c147c0563c9758bdad988ffda8d64daa2db6 - psql: Catch and report errors while printing result table. Errors (for example I/O errors or disk full) while printing out result tables were completely ignored, which could result in silently truncated output in scripts, for example. Fix by adding some basic error checking and reporting. Author: Daniel Verite <daniel@manitou-mail.org> Author: David Zhang <david.zhang@highgo.ca> Discussion: https://www.postgresql.org/message-id/flat/9a0b3c8d-ee14-4b1d-9d0a-2c993bdabacc@manitou-mail.org https://git.postgresql.org/pg/commitdiff/b03436994bcc4909dd644fd5ae6d9a9acdf30da5 Jeff Davis pushed: - Specialize MemoryContextMemAllocated(). An AllocSet doubles the size of allocated blocks (up to maxBlockSize), which means that the current block can represent half of the total allocated space for the memory context. But the free space in the current block may never have been touched, so don't count the untouched memory as allocated for the purposes of MemoryContextMemAllocated(). Discussion: https://postgr.es/m/ec63d70b668818255486a83ffadc3aec492c1f57.camel@j-davis.com https://git.postgresql.org/pg/commitdiff/e00912e11a9ec2d29274ed8a6465e81385906dc2 - Disk-based Hash Aggregation. While performing hash aggregation, track memory usage when adding new groups to a hash table. If the memory usage exceeds work_mem, enter "spill mode". In spill mode, new groups are not created in the hash table(s), but existing groups continue to be advanced if input tuples match. Tuples that would cause a new group to be created are instead spilled to a logical tape to be processed later. The tuples are spilled in a partitioned fashion. When all tuples from the outer plan are processed (either by advancing the group or spilling the tuple), finalize and emit the groups from the hash table. Then, create new batches of work from the spilled partitions, and select one of the saved batches and process it (possibly spilling recursively). Author: Jeff Davis Reviewed-by: Tomas Vondra, Adam Lee, Justin Pryzby, Taylor Vesely, Melanie Plageman Discussion: https://postgr.es/m/507ac540ec7c20136364b5272acbcd4574aa76ef.camel@j-davis.com https://git.postgresql.org/pg/commitdiff/1f39bce021540fde00990af55b4432c55ef4b3c7 - Revert "Specialize MemoryContextMemAllocated().". This reverts commit e00912e11a9ec2d29274ed8a6465e81385906dc2. https://git.postgresql.org/pg/commitdiff/2fd6a44ad5744905e9ceacef7aaf698f608e6adb Alexander Korotkov pushed: - Add new typedefs introduced in 773df883e8 to typedefs.list. https://git.postgresql.org/pg/commitdiff/45452825e5a11f8242a8012853bd2ff81e8b9620 Bruce Momjian pushed: - pg_upgrade: make get_major_server_version() err msg consistent. This patch fixes the error message in get_major_server_version() to be "could not parse version file", and uses the full file path name, rather than just the data directory path. Also, commit 4109bb5de4 added the cause of the failure to the "could not open" error message, and improved quoting. This patch backpatches the "could not open" cause to PG 12, where it was first widely used, and backpatches the quoting fix in that patch to all supported releases. Reported-by: Tom Lane Discussion: https://postgr.es/m/87pne2w98h.fsf@wibble.ilmari.org Author: Dagfinn Ilmari Mannsåker Backpatch-through: 9.5 https://git.postgresql.org/pg/commitdiff/2247a1ea5fabafc11a6deab1c6fe822c9df45bb1 - docs: use alias in WHERE clause of full text search example. The current doc query specified an alias in the FROM clause and used in it the target list, but not in the WHERE clause. Reported-by: axykon@gmail.com Discussion: https://postgr.es/m/158316348159.30450.16075357948244298217@wrigleys.postgresql.org Backpatch-through: 9.5 https://git.postgresql.org/pg/commitdiff/a6d7e9fb2816fcd23e5d1e95a52448eb9acf7f6e Andrew Dunstan pushed: - Turn off deprecated bison warnings under MSVC. These are disabled by the configure code, so this is just fixing an inconsistency in the MSVC code. Backpatch to all live branches. https://git.postgresql.org/pg/commitdiff/71c2fd0c04a5daaaab5331879e23181bb52334e1 Noah Misch pushed: - Skip WAL for new relfilenodes, under wal_level=minimal. Until now, only selected bulk operations (e.g. COPY) did this. If a given relfilenode received both a WAL-skipping COPY and a WAL-logged operation (e.g. INSERT), recovery could lose tuples from the COPY. See src/backend/access/transam/README section "Skipping WAL for New RelFileNode" for the new coding rules. Maintainers of table access methods should examine that section. To maintain data durability, just before commit, we choose between an fsync of the relfilenode and copying its contents to WAL. A new GUC, wal_skip_threshold, guides that choice. If this change slows a workload that creates small, permanent relfilenodes under wal_level=minimal, try adjusting wal_skip_threshold. Users setting a timeout on COMMIT may need to adjust that timeout, and log_min_duration_statement analysis will reflect time consumption moving to COMMIT from commands like COPY. Internally, this requires a reliable determination of whether RollbackAndReleaseCurrentSubTransaction() would unlink a relation's current relfilenode. Introduce rd_firstRelfilenodeSubid. Amend the specification of rd_createSubid such that the field is zero when a new rel has an old rd_node. Make relcache.c retain entries for certain dropped relations until end of transaction. Back-patch to 9.5 (all supported versions). This introduces a new WAL record type, XLOG_GIST_ASSIGN_LSN, without bumping XLOG_PAGE_MAGIC. As always, update standby systems before master systems. This changes sizeof(RelationData) and sizeof(IndexStmt), breaking binary compatibility for affected extensions. (The most recent commit to affect the same class of extensions was 089e4d405d0f3b94c74a2c6a54357a84a681754b.) Kyotaro Horiguchi, reviewed (in earlier, similar versions) by Robert Haas. Heikki Linnakangas and Michael Paquier implemented earlier designs that materially clarified the problem. Reviewed, in earlier designs, by Andrew Dunstan, Andres Freund, Alvaro Herrera, Tom Lane, Fujii Masao, and Simon Riggs. Reported by Martijn van Oosterhout. Discussion: https://postgr.es/m/20150702220524.GA9392@svana.org https://git.postgresql.org/pg/commitdiff/cb2fd7eac285b1b0a24eeb2b8ed4456b66c5a09f - During heap rebuild, lock any TOAST index until end of transaction. swap_relation_files() calls toast_get_valid_index() to find and lock this index, just before swapping with the rebuilt TOAST index. The latter function releases the lock before returning. Potential for mischief is low; a concurrent session can issue ALTER INDEX ... SET (fillfactor = ...), which is not alarming. Nonetheless, changing pg_class.relfilenode without a lock is unconventional. Back-patch to 9.5 (all supported versions), because another fix needs this. Discussion: https://postgr.es/m/20191226001521.GA1772687@rfd.leadboat.com https://git.postgresql.org/pg/commitdiff/e629a01f6973688f77ecff964d9d6bea9bc4b374 - In log_newpage_range(), heed forkNum and page_std arguments. The function assumed forkNum=MAIN_FORKNUM and page_std=true, ignoring the actual arguments. Existing callers passed exactly those values, so there's no live bug. Back-patch to v12, where the function first appeared, because another fix needs this. Discussion: https://postgr.es/m/20191118045434.GA1173436@rfd.leadboat.com https://git.postgresql.org/pg/commitdiff/d3e572855be1e15c7e0a6adc8db52b9fd4f71be0 - Fix cosmetic blemishes involving rd_createSubid. Remove an obsolete comment from AtEOXact_cleanup(). Restore formatting of a comment in struct RelationData, mangled by the pgindent run in commit 9af4159fce6654aa0e081b00d02bca40b978745c. Back-patch to 9.5 (all supported versions), because another fix stacks on this. https://git.postgresql.org/pg/commitdiff/d60ef94d7639b433a7bf9dd7d48aeb63194235d3 - Revert "Skip WAL for new relfilenodes, under wal_level=minimal.". This reverts commit cb2fd7eac285b1b0a24eeb2b8ed4456b66c5a09f. Per numerous buildfarm members, it was incompatible with parallel query, and a test case assumed LP64. Back-patch to 9.5 (all supported versions). Discussion: https://postgr.es/m/20200321224920.GB1763544@rfd.leadboat.com https://git.postgresql.org/pg/commitdiff/de9396326edcbe5cafc06a72016f9d715c350e0e == Pending Patches == Justin Pryzby sent in four more revisions of a patch to make pg_ls_tmpdir and friends show directories and shared filesets. James Coleman and Tomáš Vondra traded patches to implement incremental sort. Wu Hao and Asim R P sent in another revision of a patch to start the WAL receiver before the startup process replays existing WAL. Julien Rouhaud sent in another revision of a patch to expose queryid in pg_stat_activity and log_line_prefix. Mark Dilger sent in a patch to add test coverage for ALTER SEQUENCE SET SCHEMA. Justin Pryzby sent in another revision of a patch to SRFs to avoid leaking resources if not run to completion, and clean up the existing anti-pattern of freeing SRF resources. Torikoshi Atsushi sent in a patch to fix comments on SharedHotStandbyActive. Amit Langote and Justin Pryzby traded patches to preserve replica identity index across ALTER TABLE rewrite, and use get_index_isclustered in cluster.c. Dilip Kumar and Amit Kapila traded patches to move relation extension locks out of heavyweight lock manager. Atsushi Torikoshi sent in two revisions of a patch to add types to index storage parameters in the docs. Julien Rouhaud sent in three more revisions of a patch to add a pg_check_relation() function. Atsushi Torikoshi sent in a patch to fix the type in createtable storage parameters. Zeng Wenjing sent in two more revisions of a patch to implement global temporary tables. Peter Eisentraut sent in a patch to prepare to support non-tables in publications. Jeff Davis sent in a patch to make MemoryContextMemAllocated() more precise. James Coleman sent in two more revisions of a patch to show bitmap only unfetched page count to EXPLAIN. Julien Rouhaud sent in four more revisions of a patch to implement collation versioning. Julien Rouhaud sent in two more revisions of a patch to add planning counters to pg_stat_statements. Michaël Paquier sent in a patch to add a blackhole check for C++ extensions. Michail Nikolaev sent in a patch to fix a Btree BackwardScan race condition on standby during VACUUM. Sergei Kornilov sent in two more revisions of a patch to allow online changes to primary_conninfo. Thomas Munro sent in two more revisions of a patch to implement WAL prefetching. Atsushi Torikoshi sent in a patch to correctly document the types used in index storage types for CREATE VIEW and ALTER VIEW. Sergei Kornilov sent in another revision of a patch to make walreceiver use a temporary replication slot by default. Julien Rouhaud sent in three more revisions of a patch to track WAL usage. Mark Dilger sent in three revisions of a patch to add missing Object Access hook invocations. Dean Rasheed sent in two more revisions of a patch to add support for IN and @> in functional-dependency statistics use. Laurenz Albe and David Rowley traded patches to autovacuum tables that have received only inserts. Masahiko Sawada, Andres Freund, and Tom Lane traded patches to add a missing errcode() to some ereport() calls. Paul A Jungwirth sent in two more revisions of a patch to implement multiranges. Ivan Kartyshov sent in two more revisions of a patch to make async replica to wait for lsn to be replayed. Mike Palmiotto and Justin Pryzby traded patches to add and use a subprocess infrastructure. Nikita Glukhov sent in three more revisions of a patch to implement SQL/JSON functions. Pavel Stěhule sent in two more revisions of a patch to implement generic type subscripting. Nikita Glukhov sent in another revision of a patch to introduce and use opclass parameters. Movead Li sent in another revision of a patch to fix the get_bit() function for a long bytea string. Noriyoshi Shinoda and Tomáš Vondra traded patches to document the new stxstattarget column. Yuzuko Hosoya sent in another revision of a patch to fix autovacuum on partitioned tables. Álvaro Herrera sent in another revision of a patch to implement minimal logical decoding on standbys. Justin Pryzby sent in another revision of a patch to include the leader PID in logfile. Julien Rouhaud sent in a patch to fix a comment in index_concurrently_swap(). Takashi Menjo sent in two more revisions of a patch to support non-volatile WAL buffers. Fujii Masao sent in another revision of a patch to improve recovery wait event for master. Amit Langote sent in another revision of a patch to make it possible to add partitioned tables to publications. Fujii Masao sent in another revision of a patch to pg_stat_progress_basebackup to add progress reporting for pg_basebackup on the server side. Chris Bandy and Amit Kapila traded patches to add schema and table names to partition errors. Peter Eisentraut sent in a patch to drop slot's LWLock before returning from SaveSlotToPath(). Tomáš Vondra sent in two more revisions of a patch to improve estimation of OR clauses using extended statistics, and support clauses of the form Var op Var. Pavel Stěhule sent in four more revisions of a patch to implement schema variables. Hubert Zhang sent in another revision of a patch to print the physical file path when verifying a checksum fails. Fabrízio de Royes Mello sent in a patch to fix file FDW documentation for csvlog. Thomas Munro and Kyotaro HORIGUCHI traded patches to implement a shared-memory based stats collector. Tomáš Vondra and Pengzhou Tang traded patches to implement parallel grouping sets. Justin Pryzby and Amit Kapila traded patches add block number to the error context for vacuum. Masahiko Sawada sent in three more revisions of a patch to implement an internal key management system. Corey Huinker and Álvaro Herrera traded patches to add a glossary. Justin Pryzby sent in another revision of a patch to to make explain HashAggregate report bucket and memory stats. Alexey Klyukin sent in a patch to fix some issues with building cpp extensions on PostgreSQL 10+. Peter Eisentraut sent in a patch to define EXEC_BACKEND in pg_config_manual.h. Zheng Li sent in a patch to makes it possible for correlated IN/Any subquery to be transformed to a JOIN. Thomas Munro sent in another revision of a patch to add an SQL type xid8 to expose FullTransactionId to users, and introduce xid8 variants of the txid_XXX() fmgr functions. Robert Haas sent in another revision of a patch to implement backup manifests. Tom Lane sent in another revision of a patch to reduce the lan cache overhead on plpgsql expression. Bruce Momjian and Dagfinn Ilmari Mannsåker traded patches to add missing libpgport prereq in ecpg's pgtypeslib. Justin Pryzby sent in another revision of a patch to report wait event for cost-based vacuum delay, and make vacuum report time spent in cost-based delay. Floris Van Nee sent in two more revisions of a patch to implement index skip scans. Andrew Dunstan sent in another revision of a patch to add an ssl passphrase callback. Justin Pryzby sent in a patch to fix some documentation around parallel vacuum. Tom Lane sent in another revision of a patch to improve the way square roots are computed on NUMERICs.
pgsql-announce by date: