== PostgreSQL Weekly News - April 7, 2019 == - Mailing list pgsql-announce
From | David Fetter |
---|---|
Subject | == PostgreSQL Weekly News - April 7, 2019 == |
Date | |
Msg-id | 20190408014016.GA4808@fetter.org Whole thread Raw |
List | pgsql-announce |
== PostgreSQL Weekly News - April 7, 2019 == == PostgreSQL Product News == psycopg2 2.8, a Python connector for PostgreSQL, released. http://initd.org/psycopg/articles/2019/04/04/psycopg-28-released/ repmgr 4.3, a replication manager for PostgreSQL, released. https://repmgr.org/docs/4.3/release-4.3.html pgAdmin4 4.4, a web- and native GUI control center for PostgreSQL, released. https://www.pgadmin.org/docs/pgadmin4/dev/release_notes_4_4.html == PostgreSQL Jobs for April == http://archives.postgresql.org/pgsql-jobs/2019-04/ == PostgreSQL Local == The German-speaking PostgreSQL Conference 2019 will take place on May 10, 2019 in Leipzig. http://2019.pgconf.de/ PGDay.IT 2019 will take place May 16th and May 17th in Bologna, Italy. https://2019.pgday.it/en/ PGCon 2019 will take place in Ottawa on May 28-31, 2019. https://www.pgcon.org/2019 Swiss PGDay 2019 will take place in Rapperswil (near Zurich) on June 28, 2019. The CfP is open through April 18, 2019, and registration is open. http://www.pgday.ch/2019/ PostgresLondon 2019 will be July 2-3, 2019 with an optional training day on July 1. http://postgreslondon.org PGConf.Brazil 2019 is on August 1-3 2019 in São Paulo. http://pgconf.com.br The first Austrian pgDay, will take place September 6, 2019 at the Hilton Garden Inn in Wiener Neustadt. The CfP is open until April 1, 2019. https://pgday.at/en/ == PostgreSQL in the News == Planet PostgreSQL: http://planet.postgresql.org/ PostgreSQL Weekly News is brought to you this week by David Fetter Submit news and announcements by Sunday at 3:00pm PST8PDT to david@fetter.org. == Applied Patches == Tom Lane pushed: - Compute root->qual_security_level in a less random place. We can set this up once and for all in subquery_planner's initial survey of the flattened rangetable, rather than incrementally adjusting it in build_simple_rel. The previous approach made it rather hard to reason about exactly when the value would be available, and we were definitely using it in some places before the final value was computed. Noted while fooling around with Amit Langote's patch to delay creation of inheritance child rels. That didn't break this code, but it made it even more fragile, IMO. https://git.postgresql.org/pg/commitdiff/9fd4de119c609b4110eea74f00f7b9002e14aa38 - Add test case exercising formerly-unreached code in inheritance_planner. There was some debate about whether the code I'd added to remap AppendRelInfos obtained from the initial SELECT planning run is actually necessary. Add a test case demonstrating that it is. Discussion: https://postgr.es/m/23831.1553873385@sss.pgh.pa.us https://git.postgresql.org/pg/commitdiff/8fba397f0ca7b9a1fd59ab2b676c057dde4f8219 - Add volatile qualifier missed in commit 2e616dee9. Noted by Pavel Stehule Discussion: https://postgr.es/m/CAFj8pRAaGO5FX7bnP3E=mRssoK8y5T78x7jKy-vDiyS68L888Q@mail.gmail.com https://git.postgresql.org/pg/commitdiff/b2b819019f4449f2fd889ce0c39ffa81ebfaee16 - Improve documentation about our XML functionality. Add a section explaining how our XML features depart from current versions of the SQL standard. Update and clarify the descriptions of some XML functions. Chapman Flack, reviewed by Ryan Lambert Discussion: https://postgr.es/m/5BD1284C.1010305@anastigmatix.net Discussion: https://postgr.es/m/5C81F8C0.6090901@anastigmatix.net Discussion: https://postgr.es/m/CAN-V+g-6JqUQEQZ55Q3toXEN6d5Ez5uvzL4VR+8KtvJKj31taw@mail.gmail.com https://git.postgresql.org/pg/commitdiff/12d46ac392d00fa0d103f445aa4997f97029e007 - Restrict pgbench's zipfian parameter to ensure good performance. Remove the code that supported zipfian distribution parameters less than 1.0, as it had undocumented performance hazards, and it's not clear that the case is useful enough to justify either fixing or documenting those hazards. Also, since the code path for parameter > 1.0 could perform badly for values very close to 1.0, establish a minimum allowed value of 1.001. This solution seems superior to the previous vague documentation warning about small values not performing well. Fabien Coelho, per a gripe from Tomas Vondra Discussion: https://postgr.es/m/b5e172e9-ad22-48a3-86a3-589afa20e8f7@2ndquadrant.com https://git.postgresql.org/pg/commitdiff/26a76cb64072df6fa5585c2c15df39970ccdce01 - Remove now-unnecessary thread pointer arguments in pgbench. Not required after nuking the zipfian thread-local cache. Also add a comment about hazardous pointer punning in threadRun(), and avoid using "thread" to refer to the threads array as a whole. Fabien Coelho and Tom Lane, per suggestion from Alvaro Herrera Discussion: https://postgr.es/m/alpine.DEB.2.21.1904032126060.7997@lancre https://git.postgresql.org/pg/commitdiff/d8c0bd9fefa9c70a3f5613fba672fa92f08ea940 - Doc: clarify partial-index example. Jonathan Katz Discussion: https://postgr.es/m/155432280882.722.12392985690846288230@wrigleys.postgresql.org https://git.postgresql.org/pg/commitdiff/af052daec5aa282eab2d22fa3b24d6109a30c2f3 - Make queries' locking of indexes more consistent. The assertions added by commit b04aeb0a0 exposed that there are some code paths wherein the executor will try to open an index without holding any lock on it. We do have some lock on the index's table, so it seems likely that there's no fatal problem with this (for instance, the index couldn't get dropped from under us). Still, it's bad practice and we should fix it. To do so, remove the optimizations in ExecInitIndexScan and friends that tried to avoid taking a lock on an index belonging to a target relation, and just take the lock always. In non-bug cases, this will result in no additional shared-memory access, since we'll find in the local lock table that we already have a lock of the desired type; hence, no significant performance degradation should occur. Also, adjust the planner and executor so that the type of lock taken on an index is always identical to the type of lock taken for its table, by relying on the recently added RangeTblEntry.rellockmode field. This avoids some corner cases where that might not have been true before (possibly resulting in extra locking overhead), and prevents future maintenance issues from having multiple bits of logic that all needed to be in sync. In addition, this change removes all core calls to ExecRelationIsTargetRelation, which avoids a possible O(N^2) startup penalty for queries with large numbers of target relations. (We'd probably remove that function altogether, were it not that we advertise it as something that FDWs might want to use.) Also adjust some places in selfuncs.c to not take any lock on indexes they are transiently opening, since we can assume that plancat.c did that already. In passing, change gin_clean_pending_list() to take RowExclusiveLock not AccessShareLock on its target index. Although it's not clear that that's actually a bug, it seemed very strange for a function that's explicitly going to modify the index to use only AccessShareLock. David Rowley, reviewed by Julien Rouhaud and Amit Langote, a bit of further tweaking by me Discussion: https://postgr.es/m/19465.1541636036@sss.pgh.pa.us https://git.postgresql.org/pg/commitdiff/9c703c169a872d144f2f79d2fb211c82587adfa7 - Add a "SQLSTATE-only" error verbosity option to libpq and psql. This is intended for use mostly in test scripts for external tools, which could do without cross-PG-version variations in error message wording. Of course, the SQLSTATE isn't guaranteed stable either, but it should be more so than the error message text. Note: there's a bit of an ABI change for libpq here, but it seems OK because if somebody compiles against a newer version of libpq-fe.h, and then tries to pass PQERRORS_SQLSTATE to PQsetErrorVerbosity() of an older libpq library, it will be accepted and then act like PQERRORS_DEFAULT, thanks to the way the tests in pqBuildErrorMessage3 have historically been phrased. That seems acceptable. Didier Gautheron, reviewed by Dagfinn Ilmari Mannsåker Discussion: https://postgr.es/m/CAJRYxuKyj4zA+JGVrtx8OWAuBfE-_wN4sUMK4H49EuPed=mOBw@mail.gmail.com https://git.postgresql.org/pg/commitdiff/7bac3acab4d5c3f2c35aa3a7bea08411d83fd5bc - Ensure consistent name matching behavior in processSQLNamePattern(). Prior to v12, if you used a collation-sensitive regex feature in a pattern handled by processSQLNamePattern() (for instance, \d '\\w+' in psql), the behavior you got matched the database's default collation. Since commit 586b98fdf you'd usually get C-collation behavior, because the catalog "name"-type columns are now marked as COLLATE "C". Add explicit COLLATE specifications to restore the prior behavior. (Note for whoever writes the v12 release notes: the need for this shows that while 586b98fdf preserved pre-v12 behavior of "name" columns for simple comparison operators, it changed the behavior of regex operators on those columns. Although this patch fixes it for pattern matches generated by our own tools, user-written queries will still be affected. So we'd better mention this issue as a compatibility item.) Daniel Vérité Discussion: https://postgr.es/m/701e51f0-0ec0-4e70-a365-1958d66dd8d2@manitou-mail.org https://git.postgresql.org/pg/commitdiff/478cacb50e4851760db9b8e75b2256085b571965 - Use Append rather than MergeAppend for scanning ordered partitions. If we need ordered output from a scan of a partitioned table, but the ordering matches the partition ordering, then we don't need to use a MergeAppend to combine the pre-ordered per-partition scan results: a plain Append will produce the same results. This both saves useless comparison work inside the MergeAppend proper, and allows us to start returning tuples after istarting up just the first child node not all of them. However, all is not peaches and cream, because if some of the child nodes have high startup costs then there will be big discontinuities in the tuples-returned-versus-elapsed-time curve. The planner's cost model cannot handle that (yet, anyway). If we model the Append's startup cost as being just the first child's startup cost, we may drastically underestimate the cost of fetching slightly more tuples than are available from the first child. Since we've had bad experiences with over-optimistic choices of "fast start" plans for ORDER BY LIMIT queries, that seems scary. As a klugy workaround, set the startup cost estimate for an ordered Append to be the sum of its children's startup costs (as MergeAppend would). This doesn't really describe reality, but it's less likely to cause a bad plan choice than an underestimated startup cost would. In practice, the cases where we really care about this optimization will have child plans that are IndexScans with zero startup cost, so that the overly conservative estimate is still just zero. David Rowley, reviewed by Julien Rouhaud and Antonin Houska Discussion: https://postgr.es/m/CAKJS1f-hAqhPLRk_RaSFTgYxd=Tz5hA7kQ2h4-DhJufQk8TGuw@mail.gmail.com https://git.postgresql.org/pg/commitdiff/959d00e9dbe4cfcf4a63bb655ac2c29a5e579246 - Fix failures in validateForeignKeyConstraint's slow path. The foreign-key-checking loop in ATRewriteTables failed to ignore relations without storage (e.g., partitioned tables), unlike the initial loop. This accidentally worked as long as RI_Initial_Check succeeded, which it does in most practical cases (including all the ones exercised in the existing regression tests :-(). However, if that failed, as for instance when there are permissions issues, then we entered the slow fire-the-trigger-on-each-tuple path. And that would try to read from the referencing relation, and fail if it lacks storage. A second problem, recently introduced in HEAD, was that this loop had been broken by sloppy refactoring for the tableam API changes. Repair both issues, and add a regression test case so we have some coverage on this code path. Back-patch as needed to v11. (It looks like this code could do with additional bulletproofing, but let's get a working test case in place first.) Hadi Moshayedi, Tom Lane, Andres Freund Discussion: https://postgr.es/m/CAK=1=WrnNmBbe5D9sm3t0a6dnAq3cdbF1vXY816j1wsMqzC8bw@mail.gmail.com Discussion: https://postgr.es/m/19030.1554574075@sss.pgh.pa.us Discussion: https://postgr.es/m/20190325180405.jytoehuzkeozggxx%40alap3.anarazel.de https://git.postgresql.org/pg/commitdiff/46e3442c9ec858071d60a1c0fae2e9868aeaa0c8 - Avoid Python memory leaks in hstore_plpython and jsonb_plpython. Fix some places where we might fail to do Py_DECREF() on a Python object (thereby leaking it for the rest of the session). Almost all of the risks were in error-recovery paths, which we don't really expect to hit anyway. Hence, while this is definitely a bug fix, it doesn't quite seem worth back-patching. Nikita Glukhov, Michael Paquier, Tom Lane Discussion: https://postgr.es/m/28053a7d-10d8-fc23-b05c-b4749c873f63@postgrespro.ru https://git.postgresql.org/pg/commitdiff/9e360f0e8327bcf4b7c5ce957024fdff8f95f7de - Clean up side-effects of commits ab5fcf2b0 et al. Before those commits, partitioning-related code in the executor could assume that ModifyTableState.resultRelInfo[] contains only leaf partitions. However, now a fully-pruned update results in a dummy ModifyTable that references the root partitioned table, and that breaks some stuff. In v11, this led to an assertion or core dump in the tuple routing code. Fix by disabling tuple routing, since we don't need that anyway. (I chose to do that in HEAD as well for safety, even though the problem doesn't manifest in HEAD as it stands.) In v10, this confused ExecInitModifyTable's decision about whether it needed to close the root table. But we can get rid of that altogether by being smarter about where to find the root table. Note that since the referenced commits haven't shipped yet, this isn't fixing any bug the field has seen. Amit Langote, per a report from me Discussion: https://postgr.es/m/20710.1554582479@sss.pgh.pa.us https://git.postgresql.org/pg/commitdiff/159970bcadbbdefd312d75ce7ad90f19add19b40 Peter Geoghegan pushed: - Fix nbtree high key "continuescan" row compare bug. Commit 29b64d1d mishandled skipping over truncated high key attributes during row comparisons. The row comparison key matching loop would loop forever when a truncated attribute was encountered for a row compare subkey. Fix by following the example of other code in the loop: advance the current subkey, or break out of the loop when the last subkey is reached. Add test coverage for the relevant _bt_check_rowcompare() code path. The new test case is somewhat tied to nbtree implementation details, which isn't ideal, but seems unavoidable. https://git.postgresql.org/pg/commitdiff/76a39f2295ecb040f2ea052320941e1eb9b526c0 - Invalidate binary search bounds consistently. _bt_check_unique() failed to invalidate binary search bounds in the event of a live conflict following commit e5adcb78. This resulted in problems after waiting for the conflicting xact to commit or abort. The subsequent call to _bt_check_unique() would restore the initial binary search bounds, rather than starting a new search. Fix by explicitly invalidating bounds when it becomes clear that there is a live conflict that insertion will have to wait to resolve. Ashutosh Sharma, with a few additional tweaks by me. Author: Ashutosh Sharma Reported-By: Ashutosh Sharma Diagnosed-By: Ashutosh Sharma Discussion: https://postgr.es/m/CAE9k0PnQp-qr-UYKMSCzdC2FBzdE4wKP41hZrZvvP26dKLonLg@mail.gmail.com https://git.postgresql.org/pg/commitdiff/74eb2176bf3ac7a2fae1282b4f4f1d000f3e6d78 - Add test coverage for rootdescend verification. Commit c1afd175, which added support for rootdescend verification to amcheck, added only minimal regression test coverage. Address this by making sure that rootdescend verification is run on a multi-level index. In passing, simplify some of the regression tests that exercise multi-level nbtree page deletion. Both issues spotted while rereviewing coverage of the nbtree patch series using gcov. https://git.postgresql.org/pg/commitdiff/344b7e11bbaf5e11f2497b11405e63d190043cfe Andres Freund pushed: - tableam: Formatting and other minor cleanups. The superflous heapam_xlog.h includes were reported by Peter Geoghegan. https://git.postgresql.org/pg/commitdiff/4bb50236eb561f4639e75a393a5a1c9b8681acfb - tableam: sample scan. This moves sample scan support to below tableam. It's not optional as there is, in contrast to e.g. bitmap heap scans, no alternative way to perform tablesample queries. If an AM can't deal with the block based API, it will have to throw an ERROR. The tableam callbacks for this are block based, but given the current TsmRoutine interface, that seems to be required. The new interface doesn't require TsmRoutines to perform visibility checks anymore - that requires the TsmRoutine to know details about the AM, which we want to avoid. To continue to allow taking the returned number of tuples account SampleScanState now has a donetuples field (which previously e.g. existed in SystemRowsSamplerData), which is only incremented after the visibility check succeeds. Author: Andres Freund Discussion: https://postgr.es/m/20180703070645.wchpu5muyto5n647@alap3.anarazel.de https://git.postgresql.org/pg/commitdiff/73c954d24896aeb05de0f81d75e891a858e439e9 - tableam: bitmap table scan. This moves bitmap heap scan support to below an optional tableam callback. It's optional as the whole concept of bitmap heapscans is fairly block specific. This basically moves the work previously done in bitgetpage() into the new scan_bitmap_next_block callback, and the direct poking into the buffer done in BitmapHeapNext() into the new scan_bitmap_next_tuple() callback. The abstraction is currently somewhat leaky because nodeBitmapHeapscan.c's prefetching and visibilitymap based logic remains - it's likely that we'll later have to move more into the AM. But it's not trivial to do so without introducing a significant amount of code duplication between the AMs, so that's a project for later. Note that now nodeBitmapHeapscan.c and the associated node types are a bit misnamed. But it's not clear whether renaming wouldn't be a cure worse than the disease. Either way, that'd be best done in a separate commit. Author: Andres Freund Reviewed-By: Robert Haas (in an older version) Discussion: https://postgr.es/m/20180703070645.wchpu5muyto5n647@alap3.anarazel.de https://git.postgresql.org/pg/commitdiff/bfbcad478f05794e5e7ea1339e62a1d258c99b6c - tableam: Add table_finish_bulk_insert(). This replaces the previous calls of heap_sync() in places using bulk-insert. By passing in the flags used for bulk-insert the AM can decide (first at insert time and then during the finish call) which of the optimizations apply to it, and what operations are necessary to finish a bulk insert operation. Also change HEAP_INSERT_* flags to TABLE_INSERT, and rename hi_options to ti_options. These changes are made even in copy.c, which hasn't yet been converted to tableam. There's no harm in doing so. Author: Andres Freund Discussion: https://postgr.es/m/20180703070645.wchpu5muyto5n647@alap3.anarazel.de https://git.postgresql.org/pg/commitdiff/d45e40158623baacd3f36f92a670d435cc1e845f - Only allow heap in a number of contrib modules. Contrib modules pgrowlocks, pgstattuple and some functionality in pageinspect currently only supports the heap table AM. As they are all concerned with low-level details that aren't reasonably exposed via tableam, error out if invoked on a non heap relation. Author: Andres Freund Discussion: https://postgr.es/m/20180703070645.wchpu5muyto5n647@alap3.anarazel.de https://git.postgresql.org/pg/commitdiff/4b82664156c230b59607704506f5b0a32ef490a2 - tableam: basic documentation. This adds documentation about the user oriented parts of table access methods (i.e. the default_table_access_method GUC and the USING clause for CREATE TABLE etc), adds a basic chapter about the table access method interface, and adds a note to storage.sgml that it's contents don't necessarily apply for non-builtin AMs. Author: Haribabu Kommi and Andres Freund Discussion: https://postgr.es/m/20180703070645.wchpu5muyto5n647@alap3.anarazel.de https://git.postgresql.org/pg/commitdiff/b73c3a11963c8bb783993cfffabb09f558f86e37 - tableam: Add table_multi_insert() and revamp/speed-up COPY FROM buffering. This adds table_multi_insert(), and converts COPY FROM, the only user of heap_multi_insert, to it. A simple conversion of COPY FROM use slots would have yielded a slowdown when inserting into a partitioned table for some workloads. Different partitions might need different slots (both slot types and their descriptors), and dropping / creating slots when there's constant partition changes is measurable. Thus instead revamp the COPY FROM buffering for partitioned tables to allow to buffer inserts into multiple tables, flushing only when limits are reached across all partition buffers. By only dropping slots when there've been inserts into too many different partitions, the aforementioned overhead is gone. By allowing larger batches, even when there are frequent partition changes, we actuall speed such cases up significantly. By using slots COPY of very narrow rows into unlogged / temporary might slow down very slightly (due to the indirect function calls). Author: David Rowley, Andres Freund, Haribabu Kommi Discussion: https://postgr.es/m/20180703070645.wchpu5muyto5n647@alap3.anarazel.de https://postgr.es/m/20190327054923.t3epfuewxfqdt22e@alap3.anarazel.de https://git.postgresql.org/pg/commitdiff/86b85044e823a304d2a265abc030254d39efe7df - Remove unused struct member, enforce multi_insert callback presence. Author: David Rowley, Andres Freund Discussion: https://postgr.es/m/CAKJS1f9=9phmm66diAji4gvHnWSrK7BGFoNct+mEUT_c8pPOjw@mail.gmail.com https://git.postgresql.org/pg/commitdiff/57a7a3adfe4eadb471bb45a7effb1a99ebb00a7b - Harden tableam against nonexistant / wrong kind of AMs. Previously it was allowed to set default_table_access_method to an empty string. That makes sense for default_tablespace, where that was copied from, as it signals falling back to the database's default tablespace. As there is no equivalent for table AMs, forbid that. Also make sure to throw a usable error when creating a table using an index AM, by using get_am_type_oid() to implement get_table_am_oid() instead of a separate copy. Previously we'd error out only later, in GetTableAmRoutine(). Thirdly remove GetTableAmRoutineByAmId() - it was only used in an earlier version of 8586bf7ed8. Add tests for the above (some for index AMs as well). https://git.postgresql.org/pg/commitdiff/ea97e440b8570ffd1a6cd6604f2ef882c0a72291 - table: docs: fix typos and grammar. Author: Justin Pryzby Discussion: https://postgr.es/m/20190404055138.GA24864@telsasoft.com https://git.postgresql.org/pg/commitdiff/86cc06d1cf9c30be3b79207242e6746f0f0b681c Noah Misch pushed: - Update HINT for pre-existing shared memory block. One should almost always terminate an old process, not use a manual removal tool like ipcrm. Removal of the ipcclean script eleven years ago (39627b1ae680cba44f6e56ca5facec4fdbfe9495) and its non-replacement corroborate that manual shm removal is now a niche goal. Back-patch to 9.4 (all supported versions). Reviewed by Daniel Gustafsson and Kyotaro HORIGUCHI. Discussion: https://postgr.es/m/20180812064815.GB2301738@rfd.leadboat.com https://git.postgresql.org/pg/commitdiff/5a907404b52753c4d6c6a7c21765aeaa42fd6a3b - Consistently test for in-use shared memory. postmaster startup scrutinizes any shared memory segment recorded in postmaster.pid, exiting if that segment matches the current data directory and has an attached process. When the postmaster.pid file was missing, a starting postmaster used weaker checks. Change to use the same checks in both scenarios. This increases the chance of a startup failure, in lieu of data corruption, if the DBA does "kill -9 `head -n1 postmaster.pid` && rm postmaster.pid && pg_ctl -w start". A postmaster will no longer recycle segments pertaining to other data directories. That's good for production, but it's bad for integration tests that crash a postmaster and immediately delete its data directory. Such a test now leaks a segment indefinitely. No "make check-world" test does that. win32_shmem.c already avoided all these problems. In 9.6 and later, enhance PostgresNode to facilitate testing. Back-patch to 9.4 (all supported versions). Reviewed by Daniel Gustafsson and Kyotaro HORIGUCHI. Discussion: https://postgr.es/m/20130911033341.GD225735@tornado.leadboat.com https://git.postgresql.org/pg/commitdiff/2f932f71d9f2963bbd201129d7b971c8f5f077fd - Handle USE_MODULE_DB for all tests able to use an installed postmaster. When $(MODULES) and $(MODULE_big) are empty, derive the database name from the first element of $(REGRESS) instead of using a constant string. When deriving the database name from $(MODULES), use its first element instead of the entire list; the earlier approach would fail if any multi-module directory had $(REGRESS) tests. Treat isolation suites and src/pl correspondingly. Under USE_MODULE_DB=1, installcheck-world and check-world no longer reuse any database name in a given postmaster. Buildfarm members axolotl, mandrill and frogfish saw spurious "is being accessed by other users" failures that would not have happened without database name reuse. (The CountOtherDBBackends() 5s deadline expired during DROP DATABASE; a backend for an earlier test suite had used the same database name and had not yet exited.) Back-patch to 9.4 (all supported versions), except bits pertaining to isolation suites. Concept reviewed by Andrew Dunstan, Andres Freund and Tom Lane. Discussion: https://postgr.es/m/20190401135213.GE891537@rfd.leadboat.com https://git.postgresql.org/pg/commitdiff/f433394e48afbb733b1436276a5a08cd78443164 - Assert that pgwin32_signal_initialize() has been called early enough. Before the pgwin32_signal_initialize() call, the backend version of pg_usleep() has no effect. No in-tree code falls afoul of that today, but temporary commit 23078689a9921968ac0873b017be6e7f772f10bc did so. Discussion: https://postgr.es/m/20190402135442.GA1173872@rfd.leadboat.com https://git.postgresql.org/pg/commitdiff/ab9ed9be2378f02edf613e0543a29287e7484338 - Make src/test/recovery/t/017_shm.pl safe for concurrent execution. Buildfarm members idiacanthus and komodoensis, which share a host, both executed this test in the same second. That failed. Back-patch to 9.6, where the test first appeared. Discussion: https://postgr.es/m/20190404020543.GA1319573@rfd.leadboat.com https://git.postgresql.org/pg/commitdiff/16ee6eaf80a40007a138b60bb5661660058d0422 - Silence -Wimplicit-fallthrough in sysv_shmem.c. Commit 2f932f71d9f2963bbd201129d7b971c8f5f077fd added code that elicits a warning on buildfarm member flaviventris. Back-patch to 9.4, like that commit. Reported by Andres Freund. Discussion: https://postgr.es/m/20190404020057.galelv7by75ekqrh@alap3.anarazel.de https://git.postgresql.org/pg/commitdiff/6f0e190056fe441f7cf788ff19b62b13c94f68f3 - Revert "Consistently test for in-use shared memory.". This reverts commits 2f932f71d9f2963bbd201129d7b971c8f5f077fd, 16ee6eaf80a40007a138b60bb5661660058d0422 and 6f0e190056fe441f7cf788ff19b62b13c94f68f3. The buildfarm has revealed several bugs. Back-patch like the original commits. Discussion: https://postgr.es/m/20190404145319.GA1720877@rfd.leadboat.com https://git.postgresql.org/pg/commitdiff/82150a05be0dc8a8ec708d1f6e6e1ef26c0e58d4 Michaël Paquier pushed: - Fix thinko in allocation call during MVC list deserialization. Spotted by Coverity. https://git.postgresql.org/pg/commitdiff/4ae7f02b0364ccba49a82efbfff46125fb357d6c - Add progress reporting to pg_checksums. This adds a new option to pg_checksums called -P/--progress, showing every second some information about the computation state of an operation for --check and --enable (--disable only updates the control file and is quick). This requires a pre-scan of the data folder so as the total size of checksummable items can be calculated, and then it gets compared to the amount processed. Similarly to what is done for pg_rewind and pg_basebackup, the information printed in the progress report consists of the current amount of data computed and the total amount of data to compute. This could be extended later on. Author: Michael Banck, Bernd Helmle Reviewed-by: Fabien Coelho, Michael Paquier Discussion: https://postgr.es/m/1535719851.1286.17.camel@credativ.de https://git.postgresql.org/pg/commitdiff/280e5f14056bf34a0f52320f659fb93acfda0876 - Improve readability of some tests in strings.sql. c251336 has added some tests to check if a toast relation should be empty or not, hardcoding the toast relation name when calling pg_relation_size(). pg_class.reltoastrelid offers the same information, so simplify the tests to use that. Reviewed-by: Daniel Gustafsson Discussion: https://postgr.es/m/20190403065949.GH3298@paquier.xyz https://git.postgresql.org/pg/commitdiff/92c76021aed47daa61afd5c7d6f8201382dee203 - Fix some documentation in pg_rewind. Since 11, it is possible to use a non-superuser role when using an online source cluster with pg_rewind as long as the role has proper permissions to execute on the source all the functions used by pg_rewind, and the documentation stated that a superuser is necessary. Let's add at the same time all the details needed to create such a role. A second confusion which comes a lot from users is that it is necessary to issue a checkpoint on a freshly-promoted standby so as its control file has up-to-date timeline information which is used by pg_rewind to validate the operation. Let's document that properly. This is back-patched down to 9.5 where pg_rewind has been introduced. Author: Michael Paquier Reviewed-by: Magnus Hagander Discussion: https://postgr.es/m/CABUevEz5bpvbwVsYCaSMV80CBZ5-82nkMzbb+Bu=h1m=rLdn=g@mail.gmail.com Backpatch-through: 9.5 https://git.postgresql.org/pg/commitdiff/bfc80683ce51590545350b2c8ccfaa9e7a2a1870 - Add support TCP user timeout in libpq and the backend server. Similarly to the set of parameters for keepalive, a connection parameter for libpq is added as well as a backend GUC, called tcp_user_timeout. Increasing the TCP user timeout is useful to allow a connection to survive extended periods without end-to-end connection, and decreasing it allows application to fail faster. By default, the parameter is 0, which makes the connection use the system default, and follows a logic close to the keepalive parameters in its handling. When connecting through a Unix-socket domain, the parameters have no effect. Author: Ryohei Nagaura Reviewed-by: Fabien Coelho, Robert Haas, Kyotaro Horiguchi, Kirk Jamison, Mikalai Keida, Takayuki Tsunakawa, Andrei Yahorau Discussion: https://postgr.es/m/EDA4195584F5064680D8130B1CA91C45367328@G01JPEXMBYT04 https://git.postgresql.org/pg/commitdiff/249d64999615802752940e017ee5166e726bc7cd Peter Eisentraut pushed: - Catch syntax error in generated column definition. The syntax GENERATED BY DEFAULT AS (expr) is not allowed but we have to accept it in the grammar to avoid shift/reduce conflicts because of the similar syntax for identity columns. The existing code just ignored this, incorrectly. Add an explicit error check and a bespoke error message. Reported-by: Justin Pryzby <pryzby@telsasoft.com> https://git.postgresql.org/pg/commitdiff/7241911782a7420e38b6e50b57d304ea48bc5362 - Unified logging system for command-line programs. This unifies the various ad hoc logging (message printing, error printing) systems used throughout the command-line programs. Features: - Program name is automatically prefixed. - Message string does not end with newline. This removes a common source of inconsistencies and omissions. - Additionally, a final newline is automatically stripped, simplifying use of PQerrorMessage() etc., another common source of mistakes. - I converted error message strings to use %m where possible. - As a result of the above several points, more translatable message strings can be shared between different components and between frontends and backend, without gratuitous punctuation or whitespace differences. - There is support for setting a "log level". This is not meant to be user-facing, but can be used internally to implement debug or verbose modes. - Lazy argument evaluation, so no significant overhead if logging at some level is disabled. - Some color in the messages, similar to gcc and clang. Set PG_COLOR=auto to try it out. Some colors are predefined, but can be customized by setting PG_COLORS. - Common files (common/, fe_utils/, etc.) can handle logging much more simply by just using one API without worrying too much about the context of the calling program, requiring callbacks, or having to pass "progname" around everywhere. - Some programs called setvbuf() to make sure that stderr is unbuffered, even on Windows. But not all programs did that. This is now done centrally. Soft goals: - Reduces vertical space use and visual complexity of error reporting in the source code. - Encourages more deliberate classification of messages. For example, in some cases it wasn't clear without analyzing the surrounding code whether a message was meant as an error or just an info. - Concepts and terms are vaguely aligned with popular logging frameworks such as log4j and Python logging. This is all just about printing stuff out. Nothing affects program flow (e.g., fatal exits). The uses are just too varied to do that. Some existing code had wrappers that do some kind of print-and-exit, and I adapted those. I tried to keep the output mostly the same, but there is a lot of historical baggage to unwind and special cases to consider, and I might not always have succeeded. One significant change is that pg_rewind used to write all error messages to stdout. That is now changed to stderr. Reviewed-by: Donald Dong <xdong@csumb.edu> Reviewed-by: Arthur Zakirov <a.zakirov@postgrespro.ru> Discussion: https://www.postgresql.org/message-id/flat/6a609b43-4f57-7348-6480-bd022f924310@2ndquadrant.com https://git.postgresql.org/pg/commitdiff/cc8d41511721d25d557fc02a46c053c0a602fed0 - file_fdw: Fix for generated columns. Since file_fdw uses COPY internally, but COPY doesn't allow listing generated columns in its column list, we need to make sure that we don't add generated columns to the column lists internally generated by file_fdw. Reported-by: Erik Rijkers <er@xs4all.nl> https://git.postgresql.org/pg/commitdiff/33215d113d61980a4b2f2aec36cdcd21fa58c1c4 - Fix compiler warning. Rewrite get_attgenerated() to avoid compiler warning if the compiler does not recognize that elog(ERROR) does not return. Reported-by: David Rowley <david.rowley@2ndquadrant.com> https://git.postgresql.org/pg/commitdiff/edda32ee250fe480701083d4e072e06929991bf9 - Cast pg_stat_progress_cluster.cluster_index_relid to oid. It's tracked internally as bigint, but when presented to the user it should be oid. https://git.postgresql.org/pg/commitdiff/106f2eb664bbd38c83090becff9fcde1e9622c9c - Report progress of REINDEX operations. This uses the same infrastructure that the CREATE INDEX progress reporting uses. Add a column to pg_stat_progress_create_index to report the OID of the index being worked on. This was not necessary for CREATE INDEX, but it's useful for REINDEX. Also edit the phase descriptions a bit to be more consistent with the source code comments. Discussion: https://www.postgresql.org/message-id/ef6a6757-c36a-9e81-123f-13b19e36b7d7%402ndquadrant.com https://git.postgresql.org/pg/commitdiff/03f9e5cba0ee1633af4abe734504df50af46fbd8 Alexander Korotkov pushed: - GIN support for @@ and @? jsonpath operators. This commit makes existing GIN operator classes jsonb_ops and json_path_ops support "jsonb @@ jsonpath" and "jsonb @? jsonpath" operators. Basic idea is to extract statements of following form out of jsonpath. key1.key2. ... .keyN = const The rest of jsonpath is rechecked from heap. Catversion is bumped. Discussion: https://postgr.es/m/fcc6fc6a-b497-f39a-923d-aa34d0c588e8%402ndQuadrant.com Author: Nikita Glukhov, Alexander Korotkov Reviewed-by: Jonathan Katz, Pavel Stehule https://git.postgresql.org/pg/commitdiff/0a02e2ae0236103e641f6570b8135b7ee8a83686 - Restrict some cases in parsing numerics in jsonpath. Jsonpath now accepts integers with leading zeroes and floats starting with a dot. However, SQL standard requires to follow JSON specification, which doesn't allow none of these cases. Our json[b] datatypes also restrict that. So, restrict it in jsonpath altogether. Author: Nikita Glukhov https://git.postgresql.org/pg/commitdiff/2e643501e5281ad5e0fa626dab1d51c1d38f639a - Throw error in jsonb_path_match() when result is not single boolean. jsonb_path_match() checks if jsonb document matches jsonpath query. Therefore, jsonpath query should return single boolean. Currently, if result of jsonpath is not a single boolean, NULL is returned independently whether silent mode is on or off. But that appears to be wrong when silent mode is off. This commit makes jsonb_path_match() throw an error in this case. Author: Nikita Glukhov https://git.postgresql.org/pg/commitdiff/b4cc19ab01ffe6a72a915b21aa41536de80923f5 Thomas Munro pushed: - Fix deadlock in heap_compute_xid_horizon_for_tuples(). We can't call code that uses syscache while we hold buffer locks on a catalog relation. If passed such a relation, just fall back to the general effective_io_concurrency GUC rather than trying to look up the containing tablespace's IO concurrency setting. We might find a better way to control prefetching in follow-up work, but for now this is enough to avoid the deadlock introduced by commit 558a9165e0. Reviewed-by: Andres Freund Diagnosed-by: Peter Geoghegan Discussion: https://postgr.es/m/CA%2BhUKGLCwPF0S4Mk7S8qw%2BDK0Bq65LueN9rofAA3HHSYikW-Zw%40mail.gmail.com Discussion: https://postgr.es/m/962831d8-c18d-180d-75fb-8b842e3a2742%40chrullrich.net https://git.postgresql.org/pg/commitdiff/4fd05bb55b40a3c9dde2b19942f275fc31b5225a - Add wal_recycle and wal_init_zero GUCs. On at least ZFS, it can be beneficial to create new WAL files every time and not to bother zero-filling them. Since it's not clear which other filesystems might benefit from one or both of those things, add individual GUCs to control those two behaviors independently and make only very general statements in the docs. Author: Jerry Jelinek, with some adjustments by Thomas Munro Reviewed-by: Alvaro Herrera, Andres Freund, Tomas Vondra, Robert Haas and others Discussion: https://postgr.es/m/CACPQ5Fo00QR7LNAcd1ZjgoBi4y97%2BK760YABs0vQHH5dLdkkMA%40mail.gmail.com https://git.postgresql.org/pg/commitdiff/475861b2615dd63ae8431d811749a6f9a15bbfd6 - Refactor the fsync queue for wider use. Previously, md.c and checkpointer.c were tightly integrated so that fsync calls could be handed off and processed in the background. Introduce a system of callbacks and file tags, so that other modules can hand off fsync work in the same way. For now only md.c uses the new interface, but other users are being proposed. Since there may be use cases that are not strictly SMGR implementations, use a new function table for sync handlers rather than extending the traditional SMGR one. Instead of using a bitmapset of segment numbers for each RelFileNode in the checkpointer's hash table, make the segment number part of the key. This requires sending explicit "forget" requests for every segment individually when relations are dropped, but suits the file layout schemes of proposed future users better (ie sparse or high segment numbers). Author: Shawn Debnath and Thomas Munro Reviewed-by: Thomas Munro, Andres Freund Discussion: https://postgr.es/m/CAEepm=2gTANm=e3ARnJT=n0h8hf88wqmaZxk0JYkxw+b21fNrw@mail.gmail.com https://git.postgresql.org/pg/commitdiff/3eb77eba5a51780d5cf52cd66a9844cd4d26feb0 - Fix bugs in mdsyncfiletag(). Commit 3eb77eba moved a _mdfd_getseg() call from mdsync() into a new callback function mdsyncfiletag(), but didn't get the arguments quite right. Without the EXTENSION_DONT_CHECK_SIZE flag we fail to open a segment if lower-numbered segments have been truncated, and it wants a block number rather than a segment number. While comparing with the older coding, also remove an unnecessary clobbering of errno, and adjust the code in mdunlinkfiletag() to ressemble the original code from mdpostckpt() more closely instead of using an unnecessary call to smgropen(). Author: Thomas Munro Discussion: https://postgr.es/m/CA%2BhUKGL%2BYLUOA0eYiBXBfwW%2BbH5kFgh94%3DgQH0jHEJ-t5Y91wQ%40mail.gmail.com https://git.postgresql.org/pg/commitdiff/794c543b1736f71ff5bc5231eee41cdf460ad8ce - Wake up interested backends when a checkpoint fails. Commit c6c9474a switched to condition variables instead of sleep loops to notify backends of checkpoint start and stop, but forgot to broadcast in case of checkpoint failure. Author: Thomas Munro Discussion: https://postgr.es/m/CA%2BhUKGJKbCd%2B_K%2BSEBsbHxVT60SG0ivWHHAdvL0bLTUt2xpA2w%40mail.gmail.com https://git.postgresql.org/pg/commitdiff/de2b38419c4018fc8c2c917cc2cc752063096cfb Dean Rasheed pushed: - Perform RLS subquery checks as the right user when going via a view. When accessing a table with RLS via a view, the RLS checks are performed as the view owner. However, the code neglected to propagate that to any subqueries in the RLS checks. Fix that by calling setRuleCheckAsUser() for all RLS policy quals and withCheckOption checks for RTEs with RLS. Back-patch to 9.5 where RLS was added. Per bug #15708 from daurnimator. Discussion: https://postgr.es/m/15708-d65cab2ce9b1717a@postgresql.org https://git.postgresql.org/pg/commitdiff/e2d28c0f404713f564dc2250646551c75172f17b Etsuro Fujita pushed: - postgres_fdw: Perform the (ORDERED, NULL) upperrel operations remotely. The upper-planner pathification allows FDWs to arrange to push down different types of upper-stage operations to the remote side. This commit teaches postgres_fdw to do it for the (ORDERED, NULL) upperrel, which is responsible for evaluating the query's ORDER BY ordering. Since postgres_fdw is already able to evaluate that ordering remotely for foreign baserels and foreign joinrels (see commit aa09cd242f et al.), this adds support for that for foreign grouping relations. Author: Etsuro Fujita Reviewed-By: Antonin Houska and Jeff Janes Discussion: https://postgr.es/m/87pnz1aby9.fsf@news-spur.riddles.org.uk https://git.postgresql.org/pg/commitdiff/ffab494a4d4676a71e830b2d7285eeeb5f2b53d7 - postgres_fdw: Modify regression tests for EPQ-related planning problems. This prevents the tests added by commit 4bbf6edfbd and adjusted by commit 99f6a17dd6 from being useless by plan changes created by an upcoming commit. Author: Etsuro Fujita Discussion: https://postgr.es/m/87pnz1aby9.fsf@news-spur.riddles.org.uk https://git.postgresql.org/pg/commitdiff/0269edefacb78700c95572c41a5a4c8633b631b6 - Refactor create_limit_path() to share cost adjustment code with FDWs. This is in preparation for an upcoming commit. Author: Etsuro Fujita Reviewed-By: Antonin Houska and Jeff Janes Discussion: https://postgr.es/m/87pnz1aby9.fsf@news-spur.riddles.org.uk https://git.postgresql.org/pg/commitdiff/aef65db6769e3f2c855dd89edcf95a536a9ab74f - postgres_fdw: Perform the (FINAL, NULL) upperrel operations remotely. The upper-planner pathification allows FDWs to arrange to push down different types of upper-stage operations to the remote side. This commit teaches postgres_fdw to do it for the (FINAL, NULL) upperrel, which is responsible for doing LockRows, LIMIT, and/or ModifyTable. This provides the ability for postgres_fdw to handle SELECT commands so that it 1) skips the LockRows step (if any) (note that this is safe since it performs early locking) and 2) pushes down the LIMIT and/or OFFSET restrictions (if any) to the remote side. This doesn't handle the INSERT/UPDATE/DELETE cases. Author: Etsuro Fujita Reviewed-By: Antonin Houska and Jeff Janes Discussion: https://postgr.es/m/87pnz1aby9.fsf@news-spur.riddles.org.uk https://git.postgresql.org/pg/commitdiff/d50d172e517c1d2aabff3ceb3ad3113b909c5017 - Doc: Update documentation on partitioning vs. foreign tables. The limitations that it is not allowed to create/attach a foreign table as a partition of an indexed partitioned table were not documented. Reported-By: Stepan Yankevych Author: Etsuro Fujita Reviewed-By: Amit Langote Backpatch-through: 11 where partitioned index was introduced Discussion: https://postgr.es/m/1553869152.858391073.5f8m3n0x@frv53.fwdcdn.com https://git.postgresql.org/pg/commitdiff/3e6b0c472942e9a73e0fc10659dce2cba56959a7 Stephen Frost pushed: - Add support for partial TOAST decompression. When asked for a slice of a TOAST entry, decompress enough to return the slice instead of decompressing the entire object. For use cases where the slice is at, or near, the beginning of the entry, this avoids a lot of unnecessary decompression work. This changes the signature of pglz_decompress() by adding a boolean to indicate if it's ok for the call to finish before consuming all of the source or destination buffers. Author: Paul Ramsey Reviewed-By: Rafia Sabih, Darafei Praliaskouski, Regina Obe Discussion: https://postgr.es/m/CACowWR07EDm7Y4m2kbhN_jnys%3DBBf9A6768RyQdKm_%3DNpkcaWg%40mail.gmail.com https://git.postgresql.org/pg/commitdiff/4d0e994eed83c845a05da6e9a417b4efec67efaf - GSSAPI encryption support. On both the frontend and backend, prepare for GSSAPI encryption support by moving common code for error handling into a separate file. Fix a TODO for handling multiple status messages in the process. Eliminate the OIDs, which have not been needed for some time. Add frontend and backend encryption support functions. Keep the context initiation for authentication-only separate on both the frontend and backend in order to avoid concerns about changing the requested flags to include encryption support. In postmaster, pull GSSAPI authorization checking into a shared function. Also share the initiator name between the encryption and non-encryption codepaths. For HBA, add "hostgssenc" and "hostnogssenc" entries that behave similarly to their SSL counterparts. "hostgssenc" requires either "gss", "trust", or "reject" for its authentication. Similarly, add a "gssencmode" parameter to libpq. Supported values are "disable", "require", and "prefer". Notably, negotiation will only be attempted if credentials can be acquired. Move credential acquisition into its own function to support this behavior. Add a simple pg_stat_gssapi view similar to pg_stat_ssl, for monitoring if GSSAPI authentication was used, what principal was used, and if encryption is being used on the connection. Finally, add documentation for everything new, and update existing documentation on connection security. Thanks to Michael Paquier for the Windows fixes. Author: Robbie Harwood, with changes to the read/write functions by me. Reviewed in various forms and at different times by: Michael Paquier, Andres Freund, David Steele. Discussion: https://www.postgresql.org/message-id/flat/jlg1tgq1ktm.fsf@thriss.redhat.com https://git.postgresql.org/pg/commitdiff/b0b39f72b9904bcb80f97b35837ccff1578aa4b8 - Move the be_gssapi_get_* prototypes. The be_gssapi_get_* prototypes were put close to similar ones for SSL- but a bit too close since that meant they ended up only being included for SSL-enabled builds. Move those to be under ENABLE_GSS instead. Pointed out by Tom. https://git.postgresql.org/pg/commitdiff/87e16db5ebe7256712999ec690899237029b9ae7 - Handle errors during GSSAPI startup better. There was some confusion over the format of the error message returned from the server during GSSAPI startup; specifically, it was expected that a length would be returned when, in reality, at this early stage in the startup sequence, no length is returned from the server as part of an error message. Correct the client-side code for dealing with error messages sent by the server during startup by simply reading what's available into our buffer, after we've discovered it's an error message, and then reporting back what was returned. In passing, also add in documentation of the environment variable PGGSSENCMODE which was missed previously, and adjust the code to look for the PGGSSENCMODE variable (the environment variable change was missed in the prior GSSMODE -> GSSENCMODE commit). Error-handling issue discovered by Peter Eisentraut, the rest were items discovered during testing of the error handling. https://git.postgresql.org/pg/commitdiff/c46c85d4594d52fb34d36d4761bb9cfc5626f20b Álvaro Herrera pushed: - Report progress of CREATE INDEX operations. This uses the progress reporting infrastructure added by c16dc1aca5e0, adding support for CREATE INDEX and CREATE INDEX CONCURRENTLY. There are two pieces to this: one is index-AM-agnostic, and the other is AM-specific. The latter is fairly elaborate for btrees, including reportage for parallel index builds and the separate phases that btree index creation uses; other index AMs, which are much simpler in their building procedures, have simplistic reporting only, but that seems sufficient, at least for non-concurrent builds. The index-AM-agnostic part is fairly complete, providing insight into the CONCURRENTLY wait phases as well as block-based progress during the index validation table scan. (The index validation index scan requires patching each AM, which has not been included here.) Reviewers: Rahila Syed, Pavan Deolasee, Tatsuro Yamada Discussion: https://postgr.es/m/20181220220022.mg63bhk26zdpvmcj@alvherre.pgsql https://git.postgresql.org/pg/commitdiff/ab0dfc961b6a821f23d9c40c723d11380ce195a6 - Update expected output for modified catalog definition. Pilot error in previous commit https://git.postgresql.org/pg/commitdiff/11074f26bc072334ab91e5646d113f95f884bd07 - Prevent use of uninitialized variable. Per buildfarm member longfin. https://git.postgresql.org/pg/commitdiff/e8abf97af770401934a2fc4887940b76403520f0 - Correctly initialize newly added struct member. Valgrind was rightly complaining that IndexVacuumInfo->report_progress (added by commit ab0dfc961b6a) was not being initialized in some code paths. Repair. Per buildfarm member lousyjack. https://git.postgresql.org/pg/commitdiff/5f768045a1a25847a3eea59d309e28c8141aed44 - Support foreign keys that reference partitioned tables. Previously, while primary keys could be made on partitioned tables, it was not possible to define foreign keys that reference those primary keys. Now it is possible to do that. Author: Álvaro Herrera Reviewed-by: Amit Langote, Jesper Pedersen Discussion: https://postgr.es/m/20181102234158.735b3fevta63msbj@alvherre.pgsql https://git.postgresql.org/pg/commitdiff/f56f8f8da6afd8523b4d5284e02a20ed2b33ef8d - Copy name when cloning FKs recurses to partitions. We were passing a string owned by a syscache entry, which was released before recursing. Fix by pstrdup'ing the string. Per buildfarm member prion. https://git.postgresql.org/pg/commitdiff/5f6fc34af5712995026267b9ace79d06b258a0ae - Log all statements from a sample of transactions. This is useful to obtain a view of the different transaction types in an application, regardless of the durations of the statements each runs. Author: Adrien Nayrat Reviewed-by: Masahiko Sawada, Hayato Kuroda, Andres Freund https://git.postgresql.org/pg/commitdiff/799e220346f1387e823a4dbdc3b1c8c3cdc5c3e0 - Tweak docs for log_statement_sample_rate. Author: Justin Pryzby, partly after a suggestion from Masahiko Sawada Discussion: https://postgr.es/m/20190328135918.GA27808@telsasoft.com Discussion: https://postgr.es/m/CAD21AoB9+y8N4+Fan-ne-_7J5yTybPttxeVKfwUocKp4zT1vNQ@mail.gmail.com https://git.postgresql.org/pg/commitdiff/d1f04b96b99d595e80791cdb0faa9cfdde2a5afa - pg_restore: Require "-f -" to mean stdout. The previous convention that stdout was selected by default when nothing is specified was just too error-prone. After a suggestion from Andrew Gierth. Author: Euler Taveira Reviewed-by: Yoshikazu Imai, José Arthur Benetasso Villanova Discussion: https://postgr.es/m/87sgwrmhdv.fsf@news-spur.riddles.org.uk https://git.postgresql.org/pg/commitdiff/413ccaa74d9a126b042727c826e65e2844adadac - Add facility to copy replication slots. This allows the user to create duplicates of existing replication slots, either logical or physical, and even changing properties such as whether they are temporary or the output plugin used. There are multiple uses for this, such as initializing multiple replicas using the slot for one base backup; when doing investigation of logical replication issues; and to select a different output plugins. Author: Masahiko Sawada Reviewed-by: Michael Paquier, Andres Freund, Petr Jelinek Discussion: https://postgr.es/m/CAD21AoAm7XX8y_tOPP6j4Nzzch12FvA1wPqiO690RCk+uYVstg@mail.gmail.com https://git.postgresql.org/pg/commitdiff/9f06d79ef831ffa333f908f6d3debdb654292414 - psql \dP: list partitioned tables and indexes The new command lists partitioned relations (tables and/or indexes), possibly with their sizes, possibly including partitioned partitions; their parents (if not top-level); if indexes show the tables they belong to; and their descriptions. While there are various possible improvements to this, having it in this form is already a great improvement over not having any way to obtain this report. Author: Pavel Stěhule, with help from Mathias Brossard, Amit Langote and Justin Pryzby. Reviewed-by: Amit Langote, Mathias Brossard, Melanie Plageman, Michaël Paquier, Álvaro Herrera https://git.postgresql.org/pg/commitdiff/1c5d9270e339662cdd78d51d0b859d4f0a11aa91 Heikki Linnakangas pushed: - Generate less WAL during GiST, GIN and SP-GiST index build. Instead of WAL-logging every modification during the build separately, first build the index without any WAL-logging, and make a separate pass through the index at the end, to write all pages to the WAL. This significantly reduces the amount of WAL generated, and is usually also faster, despite the extra I/O needed for the extra scan through the index. WAL generated this way is also faster to replay. For GiST, the LSN-NSN interlock makes this a little tricky. All pages must be marked with a valid (i.e. non-zero) LSN, so that the parent-child LSN-NSN interlock works correctly. We now use magic value 1 for that during index build. Change the fake LSN counter to begin from 1000, so that 1 is safely smaller than any real or fake LSN. 2 would've been enough for our purposes, but let's reserve a bigger range, in case we need more special values in the future. Author: Anastasia Lubennikova, Andrey V. Lepikhov Reviewed-by: Heikki Linnakangas, Dmitry Dolgov https://git.postgresql.org/pg/commitdiff/9155580fd5fc2a0cbb23376dfca7cd21f59c2c7b Tomáš Vondra pushed: - Reduce overhead of pg_mcv_list (de)serialization. Commit ea4e1c0e8f resolved issues with memory alignment in serialized pg_mcv_list values, but it required copying data to/from the varlena buffer during serialization and deserialization. As the MCV lits may be fairly large, the overhead (memory consumption, CPU usage) can get rather significant too. This change tweaks the serialization format so that the alignment is correct with respect to the varlena value, and so the parts may be accessed directly without copying the data. Catversion bump, as it affects existing pg_statistic_ext data. https://git.postgresql.org/pg/commitdiff/c50b3158bfd7fc8c2a0b2d2ede4415fd9859acd1 - Add SETTINGS option to EXPLAIN, to print modified settings. Query planning is affected by a number of configuration options, and it may be crucial to know which of those options were set to non-default values. With this patch you can say EXPLAIN (SETTINGS ON) to include that information in the query plan. Only options affecting planning, with values different from the built-in default are printed. This patch also adds auto_explain.log_settings option, providing the same capability in auto_explain module. Author: Tomas Vondra Reviewed-by: Rafia Sabih, John Naylor Discussion: https://postgr.es/m/e1791b4c-df9c-be02-edc5-7c8874944be0@2ndquadrant.com https://git.postgresql.org/pg/commitdiff/ea569d64ac7174d3fe657e3e682d11053ecf1866 Robert Haas pushed: - Allow VACUUM to be run with index cleanup disabled. This commit adds a new reloption, vacuum_index_cleanup, which controls whether index cleanup is performed for a particular relation by default. It also adds a new option to the VACUUM command, INDEX_CLEANUP, which can be used to override the reloption. If neither the reloption nor the VACUUM option is used, the default is true, as before. Masahiko Sawada, reviewed and tested by Nathan Bossart, Alvaro Herrera, Kyotaro Horiguchi, Darafei Praliaskouski, and me. The wording of the documentation is mostly due to me. Discussion: http://postgr.es/m/CAD21AoAt5R3DNUZSjOoXDUY=naYPUOuffVsRzuTYMz29yLzQCA@mail.gmail.com https://git.postgresql.org/pg/commitdiff/a96c41feec6b6616eb9d5baee9a9e08c20533c38 - Fix missing word. Nathan Bossart Discussion: http://postgr.es/m/2C63765B-AD31-4F6C-8DA7-C8544634C714@amazon.com https://git.postgresql.org/pg/commitdiff/6665305e17edff3e2c43d71396ea5b6a69e7b6f3 == Pending Patches == Raymond Martin sent in two more revisions of a patch to make pg_stat_statements more efficient. Amit Langote and Tom Lane traded patches to speed up planning with partitions. Surafel Temesgen and Tomáš Vondra traded patches to implement FETCH FIRST ... WITH TIES Peter Geoghegan sent in a patch to test the recent patch to make all B-Trees unique by making TID part of the comparisons. Pavel Stěhule sent in another revision of a patch to add type info support functions for functions that use "any" type. Pavel Stěhule sent in another revision of a patch to add schema variables. Kyotaro HORIGUCHI sent in two more revisions of a patch to fix the WAL-skipping feature. Yuzuko Hosoya sent in two more revisions of a patch to fix a problem with default partition pruning. Noah Misch sent in a patch to fix some mysterious crashes on Windows. Julien Rouhaud sent in three more revisions of a patch to fix a checksum error in pg_stat_databases. Liu Huailing sent in a patch to fix a memory overflow in the ECPG preproc module. Konstantin Knizhnik sent in another revision of a patch to add autoprepare. Nikolay Shaplov sent in another revision of a patch to add a dummy index am. Pavan Deolasee sent in another revision of a patch to implement COPY ... FREEZE. Liu Huailing sent in a patch to fix some misspellings. Amit Khandekar sent in two more revisions of a patch to implement minimal logical decoding on standbys. Michaël Paquier sent in another revision of a patch to add a separate table-level option to control compression. Haribabu Kommi and Amit Kapila traded patches to avoid counting transaction stats for parallel worker cooperating transaction. Kyotaro HORIGUCHI sent in a patch to fix a confusing message when starting standby without setting restore_command. Konstantin Knizhnik sent in a patch to test a combination of inheritance, invalidations, and prepared statements. Daniel Gustafsson sent in a patch to ensure that binaries used in pg_upgrade are all the same minor version. Dmitry Dolgov sent in another revision of a patch to optimize GROUP BY. Heikki Linnakangas sent in a patch to refactor the checks for deleted GiST pages and use the full 64-bit XID for checking if a deleted GiST page is old enough. Kyotaro HORIGUCHI sent in three more revisions of a patch to protect syscache from bloating with negative cache entries. Michaël Paquier sent in a patch to fix a bug in toast tuple target. Masahiko Sawada sent in three more revisions of a patch to implement block-level parallel vacuum. Daniel Vérité sent in a patch to fix an infelicity introduced by making the collation of the catalogs 'C'. Justin Pryzby sent in another revision of a patch to fix query logging of prepared statements. Takayuki Tsunakawa and Fujii Masao traded patches to add a reloption to prevent VACUUM from truncating empty pages at the end of relation. Iwata Aya sent in another revision of a patch to implement a libpq debug log. Anastasia Lubennikova sent in a patch to add a test to reproduce gist index problem caused by patch Generate less WAL during GiST, GIN and SP-GiST index build. Artur Zakirov sent in another revision of a patch to implement shared ispell dictionaries. Andres Freund sent in another revision of a patch to convert GiST to compute page level XID horizon on the primary. Nikolay Shaplov sent in another revision of a patch to improve the Ltree syntax. Pierre Ducroquet sent in a patch to use a different permission check path for indexes and relations in pg_stats. Fabien COELHO sent in a patch to add some more control on the initialization phase of pgbench. Jose Luis Tallon sent in two revisions of a patch to implement uuid_version(). Fabien COELHO sent in two revisions of a patch to pgbench to add minimal stats on initialization. Ramanarayana sent in another revision of a patch to fix a problem during Windows service start. Noah Misch sent in a patch to augment every test postgresql.conf. David Rowley sent in three more revisions of a patch to shrink a bloated locallocktable. Julien Rouhaud sent in a patch to remove some trailing whitespace in the documentation. Ramanarayana sent in another revision of a patch to add Perldoc for TestLib module in Postgres. Thomas Munro sent in a patch to fix pgindent's postprocessing of multi-line prototypes. Justin Pryzby sent in another revision of a patch to clean up pg_checksums.sgml. Justin Pryzby sent in a patch to clean up/remove/update references to OID column. Andres Freund sent in a PoC patch to fix an infelicity between COPY FREEZE and setting PD_ALL_VISIBLE/visibility map bits.
pgsql-announce by date: