PostgreSQL Weekly News - February 7, 2021PostgreSQL Weekly News - February 7, 2021pg_probackup 2.4.9, a utility to manage backup and recovery of PostgreSQL database clusters, released. https://github.com/postgrespro/pg_probackup/releases/tag/2.4.9 pitrery 3.3, a set of Bash scripts to manage PITR backups for PostgreSQL, released. http://dalibo.github.io/pitrery/ Person of the week: https://postgresql.life/post/alexander_sosna/ PostgreSQL Product NewsPostgreSQL Jobs for Februaryhttps://archives.postgresql.org/pgsql-jobs/2021-02/ PostgreSQL in the NewsPlanet PostgreSQL: https://planet.postgresql.org/ PostgreSQL Weekly News is brought to you this week by David Fetter Submit news and announcements by Sunday at 3:00pm PST8PDT to david@fetter.org. Applied PatchesAlexander Korotkov pushed: Implementation of subscripting for jsonb. Subscripting for jsonb does not support slices, does not have a limit for the number of subscripts, and an assignment expects a replace value to have jsonb type. There is also one functional difference between assignment via subscripting and assignment via jsonb_set(). When an original jsonb container is NULL, the subscripting replaces it with an empty jsonb and proceeds with an assignment. For the sake of code reuse, we rearrange some parts of jsonb functionality to allow the usage of the same functions for jsonb_set and assign subscripting operation. The original idea belongs to Oleg Bartunov. Catversion is bumped. Discussion: https://postgr.es/m/CA%2Bq6zcV8qvGcDXurwwgUbwACV86Th7G80pnubg42e-p9gsSf%3Dg%40mail.gmail.com Discussion: https://postgr.es/m/CA%2Bq6zcX3mdxGCgdThzuySwH-ApyHHM-G4oB1R0fn0j2hZqqkLQ%40mail.gmail.com Discussion: https://postgr.es/m/CA%2Bq6zcVDuGBv%3DM0FqBYX8DPebS3F_0KQ6OVFobGJPM507_SZ_w%40mail.gmail.com Discussion: https://postgr.es/m/CA%2Bq6zcVovR%2BXY4mfk-7oNk-rF91gH0PebnNfuUjuuDsyHjOcVA%40mail.gmail.com Author: Dmitry Dolgov Reviewed-by: Tom Lane, Arthur Zakirov, Pavel Stehule, Dian M Fay Reviewed-by: Andrew Dunstan, Chapman Flack, Merlin Moncure, Peter Geoghegan Reviewed-by: Alvaro Herrera, Jim Nasby, Josh Berkus, Victor Wagner Reviewed-by: Aleksander Alekseev, Robert Haas, Oleg Bartunov https://git.postgresql.org/pg/commitdiff/676887a3b0b8e3c0348ac3f82ab0d16e9a24bd43 Filling array gaps during jsonb subscripting. This commit introduces two new flags for jsonb assignment: * JB_PATH_FILL_GAPS: Appending array elements on the specified position, gaps are filled with nulls (similar to the JavaScript behavior). This mode also instructs to create the whole path in a jsonb object if some part of the path (more than just the last element) is not present. * JB_PATH_CONSISTENT_POSITION: Assigning keeps array positions consistent by preventing prepending of elements. Both flags are used only in jsonb subscripting assignment. Initially proposed by Nikita Glukhov based on polymorphic subscripting patch, but transformed into an independent change. Discussion: https://postgr.es/m/CA%2Bq6zcV8qvGcDXurwwgUbwACV86Th7G80pnubg42e-p9gsSf%3Dg%40mail.gmail.com Discussion: https://postgr.es/m/CA%2Bq6zcX3mdxGCgdThzuySwH-ApyHHM-G4oB1R0fn0j2hZqqkLQ%40mail.gmail.com Discussion: https://postgr.es/m/CA%2Bq6zcVDuGBv%3DM0FqBYX8DPebS3F_0KQ6OVFobGJPM507_SZ_w%40mail.gmail.com Discussion: https://postgr.es/m/CA%2Bq6zcVovR%2BXY4mfk-7oNk-rF91gH0PebnNfuUjuuDsyHjOcVA%40mail.gmail.com Author: Dmitry Dolgov Reviewed-by: Tom Lane, Arthur Zakirov, Pavel Stehule, Dian M Fay Reviewed-by: Andrew Dunstan, Chapman Flack, Merlin Moncure, Peter Geoghegan Reviewed-by: Alvaro Herrera, Jim Nasby, Josh Berkus, Victor Wagner Reviewed-by: Aleksander Alekseev, Robert Haas, Oleg Bartunov https://git.postgresql.org/pg/commitdiff/81fcc72e66222357f9bccce3eeda62eb2cb29849 Throw error when assigning jsonb scalar instead of a composite object. During the jsonb subscripting assignment, the provided path might assume an object or an array where the source jsonb has a scalar value. Initial subscripting assignment logic will skip such an update operation with no message shown. This commit makes it throw an error to indicate this type of situation. Discussion: https://postgr.es/m/CA%2Bq6zcV8qvGcDXurwwgUbwACV86Th7G80pnubg42e-p9gsSf%3Dg%40mail.gmail.com Discussion: https://postgr.es/m/CA%2Bq6zcX3mdxGCgdThzuySwH-ApyHHM-G4oB1R0fn0j2hZqqkLQ%40mail.gmail.com Discussion: https://postgr.es/m/CA%2Bq6zcVDuGBv%3DM0FqBYX8DPebS3F_0KQ6OVFobGJPM507_SZ_w%40mail.gmail.com Discussion: https://postgr.es/m/CA%2Bq6zcVovR%2BXY4mfk-7oNk-rF91gH0PebnNfuUjuuDsyHjOcVA%40mail.gmail.com Author: Dmitry Dolgov Reviewed-by: Tom Lane, Arthur Zakirov, Pavel Stehule, Dian M Fay Reviewed-by: Andrew Dunstan, Chapman Flack, Merlin Moncure, Peter Geoghegan Reviewed-by: Alvaro Herrera, Jim Nasby, Josh Berkus, Victor Wagner Reviewed-by: Aleksander Alekseev, Robert Haas, Oleg Bartunov https://git.postgresql.org/pg/commitdiff/aa6e46daf5304e8d9e66fefc1a5bd77622ec6402 Get rid of unnecessary memory allocation in jsonb_subscript_assign(). Current code allocates memory for JsonbValue, but it could be placed locally. https://git.postgresql.org/pg/commitdiff/bb513b364b4fe31574574c8d0afbb2255268b321
Tom Lane pushed: Fix portability issue in new jsonbsubs code. On machines where sizeof(Datum) > sizeof(Oid) (that is, any 64-bit platform), the previous coding would compute a misaligned workspace->index pointer if nupper is odd. Architectures where misaligned access is a hard no-no would then fail. This appears to explain why thorntail is unhappy but other buildfarm members are not. https://git.postgresql.org/pg/commitdiff/7c5d57caed4d8af705d0cc3131d0d8ed72b7a41d Revise make_partition_pruneinfo to not use its partitioned_rels input. It turns out that the calculation of [Merge]AppendPath.partitioned_rels in allpaths.c is faulty and sometimes omits relevant non-leaf partitions, allowing an assertion added by commit a929e17e5a8 to trigger. Rather than fix that, it seems better to get rid of those fields altogether. We don't really need the info until create_plan time, and calculating it once for the selected plan should be cheaper than calculating it for each append path we consider. As a first step, teach make_partition_pruneinfo to collect the relevant partitioned tables for itself. It's not hard to do so by traversing from child tables up to parents using the AppendRelInfo links. While here, make some minor stylistic improvements; mainly, don't use the "Relids" alias for bitmapsets that are not identities of any relation considered by the planner. Try to document the logic better, too. No backpatch, as there does not seem to be a live problem before a929e17e5a8. Also no new regression test; the code where the bug was will be gone at the end of this patch series, so it seems a bit pointless to memorialize the issue. Tom Lane and David Rowley, per reports from Andreas Seltenreich and Jaime Casanova. Discussion: https://postgr.es/m/87sg8tqhsl.fsf@aurora.ydns.eu Discussion: https://postgr.es/m/CAJKUy5gCXDSmFs2c=R+VGgn7FiYcLCsEFEuDNNLGfoha=pBE_g@mail.gmail.com https://git.postgresql.org/pg/commitdiff/fb2d645dd53ff571572d830e830fc8c368063802 Remove incidental dependencies on partitioned_rels lists. It turns out that the calculation of [Merge]AppendPath.partitioned_rels in allpaths.c is faulty and sometimes omits relevant non-leaf partitions, allowing an assertion added by commit a929e17e5a8 to trigger. Rather than fix that, it seems better to get rid of those fields altogether. We don't really need the info until create_plan time, and calculating it once for the selected plan should be cheaper than calculating it for each append path we consider. This patch undoes a couple of very minor uses of the partitioned_rels values. createplan.c was testing for nil-ness to optimize away the preparatory work for make_partition_pruneinfo(). That is worth doing if the check is nigh free, but it's not worth going to any great lengths to avoid. create_append_path() was testing for nil-ness as part of deciding how to set up ParamPathInfo for an AppendPath. I replaced that with a check for the appendrel's parent rel being partitioned. That's not quite the same thing but should cover most cases. If we note any interesting loss of optimizations, we can dumb this down to just always use the more expensive method when the parent is a baserel. Discussion: https://postgr.es/m/87sg8tqhsl.fsf@aurora.ydns.eu Discussion: https://postgr.es/m/CAJKUy5gCXDSmFs2c=R+VGgn7FiYcLCsEFEuDNNLGfoha=pBE_g@mail.gmail.com https://git.postgresql.org/pg/commitdiff/5076f88bc985a7728eea337cbabae0e034b064b1 Remove [Merge]AppendPath.partitioned_rels. It turns out that the calculation of [Merge]AppendPath.partitioned_rels in allpaths.c is faulty and sometimes omits relevant non-leaf partitions, allowing an assertion added by commit a929e17e5a8 to trigger. Rather than fix that, it seems better to get rid of those fields altogether. We don't really need the info until create_plan time, and calculating it once for the selected plan should be cheaper than calculating it for each append path we consider. The preceding two commits did away with all use of the partitioned_rels values; this commit just mechanically removes the fields and the code that calculated them. Discussion: https://postgr.es/m/87sg8tqhsl.fsf@aurora.ydns.eu Discussion: https://postgr.es/m/CAJKUy5gCXDSmFs2c=R+VGgn7FiYcLCsEFEuDNNLGfoha=pBE_g@mail.gmail.com https://git.postgresql.org/pg/commitdiff/f003a7522bfa11177dc52c65eb97273a1057dfba Doc: work a little harder on the initial examples for regex matching. Writing unnecessary '.*' at start and end of a POSIX regex doesn't do much except confuse the reader about whether that might be necessary after all. Make the examples in table 9.16 a tad more realistic, and try to turn the next group of examples into something self-contained. Per gripe from rmzgrimes. Back-patch to v13 because it's easy. Discussion: https://postgr.es/m/161215841824.14653.8969016349304314299@wrigleys.postgresql.org https://git.postgresql.org/pg/commitdiff/9522085ac917af66dba29939af328ae67300f10a Fix ancient memory leak in contrib/auto_explain. The ExecutorEnd hook is invoked in a context that could be quite long-lived, not the executor's own per-query context as I think we were sort of assuming. Thus, any cruft generated while producing the EXPLAIN output could accumulate over multiple queries. This can result in spectacular leakage if log_nested_statements is on, and even without that I'm surprised nobody complained before. To fix, just switch into the executor's context so that anything we allocate will be released when standard_ExecutorEnd frees the executor state. We might as well nuke the code's retail pfree of the explain output string, too; that's laughably inadequate to the need. Japin Li, per report from Jeff Janes. This bug is old, so back-patch to all supported branches. Discussion: https://postgr.es/m/CAMkU=1wCVtbeRn0s9gt12KwQ7PLXovbpM8eg25SYocKW3BT4hg@mail.gmail.com https://git.postgresql.org/pg/commitdiff/5c0f7cc5442108e113d4fb88c952329b467e2c6a Remove extra increment of plpgsql's statement counter for FOR loops. This left gaps in the internal statement numbering, which is not terribly harmful (else we'd have noticed sooner), but it's not great either. Oversight in bbd5c207b; backpatch to v12 where that came in. Pavel Stehule Discussion: https://postgr.es/m/CAFj8pRDXyQaJmpotNTQVc-t-WxdWZC35V2PnmwOaV1-taidFWA@mail.gmail.com https://git.postgresql.org/pg/commitdiff/dfcc46fe3030b0114b7a5715d5fa40819561c04b Doc: consistently identify OID catalog columns that can be zero. Not all OID-reference columns that can contain zeroes (indicating "no reference") were explicitly marked in catalogs.sgml. Fix that, and try to make the style a little more consistent while at it --- for example, consistently write "zero" not "0" for these cases. Joel Jacobson and Tom Lane Discussion: https://postgr.es/m/4ed9a372-7bf9-479a-926c-ae8e774717a8@www.fastmail.com https://git.postgresql.org/pg/commitdiff/479331406e8403cc2e75d1082f8c613e7669c113 Build in some knowledge about foreign-key relationships in the catalogs. This follows in the spirit of commit dfb75e478, which created primary key and uniqueness constraints to improve the visibility of constraints imposed on the system catalogs. While our catalogs contain many foreign-key-like relationships, they don't quite follow SQL semantics, in that the convention for an omitted reference is to write zero not NULL. Plus, we have some cases in which there are arrays each of whose elements is supposed to be an FK reference; SQL has no way to model that. So we can't create actual foreign key constraints to describe the situation. Nonetheless, we can collect and use knowledge about these relationships. This patch therefore adds annotations to the catalog header files to declare foreign-key relationships. (The BKI_LOOKUP annotations cover simple cases, but we weren't previously distinguishing which such columns are allowed to contain zeroes; we also need new markings for multi-column FK references.) Then, Catalog.pm and genbki.pl are taught to collect this information into a table in a new generated header "system_fk_info.h". The only user of that at the moment is a new SQL function pg_get_catalog_foreign_keys(), which exposes the table to SQL. The oidjoins regression test is rewritten to use pg_get_catalog_foreign_keys() to find out which columns to check. Aside from removing the need for manual maintenance of that test script, this allows it to cover numerous relationships that were not checked by the old implementation based on findoidjoins. (As of this commit, 217 relationships are checked by the test, versus 181 before.) Discussion: https://postgr.es/m/3240355.1612129197@sss.pgh.pa.us https://git.postgresql.org/pg/commitdiff/62f34097c88433ef1f3de604714fe7e7024f2fdf Retire findoidjoins. In the wake of commit 62f34097c, we no longer need this tool. Discussion: https://postgr.es/m/3240355.1612129197@sss.pgh.pa.us https://git.postgresql.org/pg/commitdiff/ef3d4613c0204ab2b87ffa7e8e9551d74f932816 Remove special BKI_LOOKUP magic for namespace and role OIDs. Now that commit 62f34097c attached BKI_LOOKUP annotation to all the namespace and role OID columns in the catalogs, there's no real reason to have the magic PGNSP and PGUID symbols. Get rid of them in favor of implementing those lookups according to genbki.pl's normal pattern. This means that in the catalog headers, BKI_DEFAULT(PGNSP) becomes BKI_DEFAULT(pg_catalog), which seems a lot more transparent. BKI_DEFAULT(PGUID) becomes BKI_DEFAULT(POSTGRES), which is perhaps less so; but you can look into pg_authid.dat to discover that POSTGRES is the nonce name for the bootstrap superuser. This change also means that if we ever need cross-references in the initial catalog data to any of the other built-in roles besides POSTGRES, or to some other built-in schema besides pg_catalog, we can just do it. No catversion bump here, as there's no actual change in the contents of postgres.bki. Discussion: https://postgr.es/m/3240355.1612129197@sss.pgh.pa.us https://git.postgresql.org/pg/commitdiff/ba0faf81c65ac99dd42ce192f3257d4d2231ea50 Avoid crash when rolling back within a prepared statement. If a portal is used to run a prepared CALL or DO statement that contains a ROLLBACK, PortalRunMulti fails because the portal's statement list gets cleared by the rollback. (Since the grammar doesn't allow CALL/DO in PREPARE, the only easy way to get to this is via extended query protocol, which treats all inputs as prepared statements.) It's difficult to avoid resetting the portal early because of resource-management issues, so work around this by teaching PortalRunMulti to be wary of portal->stmts having suddenly become NIL. The crash has only been seen to occur in v13 and HEAD (as a consequence of commit 1cff1b95a having added an extra touch of portal->stmts). But even before that, the code involved touching a List that the portal no longer has any claim on. In the test case at hand, the List will still exist because of another refcount on the cached plan; but I'm far from convinced that it's impossible for the cached plan to have been dropped by the time control gets back to PortalRunMulti. Hence, backpatch to v11 where nested transactions were added. Thomas Munro and Tom Lane, per bug #16811 from James Inform Discussion: https://postgr.es/m/16811-c1b599b2c6c2d622@postgresql.org https://git.postgresql.org/pg/commitdiff/9624321ec502f4e4f4722290b358694049447f95 Fix YA incremental sort bug. switchToPresortedPrefixMode() did the wrong thing if it detected a batch boundary just at the last tuple of a fullsort group. The initially-reported symptom was a "retrieved too many tuples in a bounded sort" error, but the test case added here just silently gives the wrong answer without this patch. I (tgl) am not really happy about committing this patch without review from the incremental-sort authors, but they seem AWOL and we are hard against a release deadline. This does demonstrably make some cases better, anyway. Per bug #16846 from Yoran Heling. Back-patch to v13 where incremental sort was introduced. Neil Chen Discussion: https://postgr.es/m/16846-ae49f51ac379a4cb@postgresql.org https://git.postgresql.org/pg/commitdiff/82e0e29308dedbc6000e73329beb112ae7e1ad39 Fix bug in HashAgg's selective-column-spilling logic. Commit 230230223 taught nodeAgg.c that, when spilling tuples from memory in an oversized hash aggregation, it only needed to spill input columns referenced in the node's tlist and quals. Unfortunately, that's wrong: we also have to save the grouping columns. The error is masked in common cases because the grouping columns also appear in the tlist, but that's not necessarily true. The main category of plans where it's not true seem to come from semijoins ("WHERE outercol IN (SELECT innercol FROM innertable)") where the innercol needs an implicit promotion to make it comparable to the outercol. The grouping column will be "innercol::promotedtype", but that expression appears nowhere in the Agg node's own tlist and quals; only the bare "innercol" is found in the tlist. I spent quite a bit of time looking for a suitable regression test case for this, without much success. If the number of distinct values of the innercol is large enough to make spilling happen, the planner tends to prefer a non-HashAgg plan, at least for problem sizes that are reasonable to use in the regression tests. So, no new regression test. However, this patch does demonstrably fix the originally-reported test case. Per report from s.p.e (at) gmx-topmail.de. Backpatch to v13 where the troublesome code came in. Discussion: https://postgr.es/m/trinity-1c565d44-159f-488b-a518-caf13883134f-1611835701633@3c-app-gmx-bap78 https://git.postgresql.org/pg/commitdiff/0ff865fbe50e82f17df8a9280fa01faf270b7f3f Disallow converting an inheritance child table to a view. Generally, members of inheritance trees must be plain tables (or, in more recent versions, foreign tables). ALTER TABLE INHERIT rejects creating an inheritance relationship that has a view at either end. When DefineQueryRewrite attempts to convert a relation to a view, it already had checks prohibiting doing so for partitioning parents or children as well as traditional-inheritance parents ... but it neglected to check that a traditional-inheritance child wasn't being converted. Since the planner assumes that any inheritance child is a table, this led to making plans that tried to do a physical scan on a view, causing failures (or even crashes, in recent versions). One could imagine trying to support such a case by expanding the view normally, but since the rewriter runs before the planner does inheritance expansion, it would take some very fundamental refactoring to make that possible. There are probably a lot of other parts of the system that don't cope well with such a situation, too. For now, just forbid it. Per bug #16856 from Yang Lin. Back-patch to all supported branches. (In versions before v10, this includes back-patching the portion of commit 501ed02cf that added has_superclass(). Perhaps the lack of that infrastructure partially explains the missing check.) Discussion: https://postgr.es/m/16856-0363e05c6e1612fd@postgresql.org https://git.postgresql.org/pg/commitdiff/dd705a039f6cd41921529fa4e971d70b224be052 Propagate CTE property flags when copying a CTE list into a rule. rewriteRuleAction() neglected this step, although it was careful to propagate other similar flags such as hasSubLinks or hasRowSecurity. Omitting to transfer hasRecursive is just cosmetic at the moment, but omitting hasModifyingCTE is a live bug, since the executor certainly looks at that. The proposed test case only fails back to v10, but since the executor examines hasModifyingCTE in 9.x as well, I suspect that a test case could be devised that fails in older branches. Given the nearness of the release deadline, though, I'm not going to spend time looking for a better test. Report and patch by Greg Nancarrow, cosmetic changes by me Discussion: https://postgr.es/m/CAJcOf-fAdj=nDKMsRhQzndm-O13NY4dL6xGcEvdX5Xvbbi0V7g@mail.gmail.com https://git.postgresql.org/pg/commitdiff/ed290896335414c6c069b9ccae1f3dcdd2fac6ba Revert "Propagate CTE property flags when copying a CTE list into a rule.". This reverts commit ed290896335414c6c069b9ccae1f3dcdd2fac6ba and equivalent back-branch commits. The issue is subtler than I thought, and it's far from new, so just before a release deadline is no time to be fooling with it. We'll consider what to do at a bit more leisure. Discussion: https://postgr.es/m/CAJcOf-fAdj=nDKMsRhQzndm-O13NY4dL6xGcEvdX5Xvbbi0V7g@mail.gmail.com https://git.postgresql.org/pg/commitdiff/d1d2979852538d7021cc809a40ef127d59747697
Michaël Paquier pushed: Introduce --with-ssl={openssl} as a configure option. This is a replacement for the existing --with-openssl, extending the logic to make easier the addition of new SSL libraries. The grammar is chosen to be similar to --with-uuid, where multiple values can be chosen, with "openssl" as the only supported value for now. The original switch, --with-openssl, is kept for compatibility. Author: Daniel Gustafsson, Michael Paquier Reviewed-by: Jacob Champion Discussion: https://postgr.es/m/FAB21FC8-0F62-434F-AA78-6BD9336D630A@yesql.se https://git.postgresql.org/pg/commitdiff/fe61df7f82aa6e0879476146dbe1da9c89b4946b Remove unused column atttypmod from initial tablesync query. The initial tablesync done by logical replication used a query to fetch the information of a relation's columns that included atttypmod, but it was left unused. This was added by 7c4f524. Author: Euler Taveira Reviewed-by: Önder Kalacı, Amit Langote, Japin Li Discussion: https://postgr.es/m/CAHE3wggb715X+mK_DitLXF25B=jE6xyNCH4YOwM860JR7HarGQ@mail.gmail.com https://git.postgresql.org/pg/commitdiff/4ad31bb2ef2517b6e49ea9e8f01aaed250d4df38 Add TABLESPACE option to REINDEX. This patch adds the possibility to move indexes to a new tablespace while rebuilding them. Both the concurrent and the non-concurrent cases are supported, and the following set of restrictions apply: - When using TABLESPACE with a REINDEX command that targets a partitioned table or index, all the indexes of the leaf partitions are moved to the new tablespace. The tablespace references of the non-leaf, partitioned tables in pg_class.reltablespace are not changed. This requires an extra ALTER TABLE SET TABLESPACE. - Any index on a toast table rebuilt as part of a parent table is kept in its original tablespace. - The operation is forbidden on system catalogs, including trying to directly move a toast relation with REINDEX. This results in an error if doing REINDEX on a single object. REINDEX SCHEMA, DATABASE and SYSTEM skip system relations when TABLESPACE is used. Author: Alexey Kondratov, Michael Paquier, Justin Pryzby Reviewed-by: Álvaro Herrera, Michael Paquier Discussion: https://postgr.es/m/8a8f5f73-00d3-55f8-7583-1375ca8f6a91@postgrespro.ru https://git.postgresql.org/pg/commitdiff/c5b286047cd698021e57a527215b48865fd4ad4e Clarify comment in tablesync.c. Author: Peter Smith Reviewed-by: Amit Kapila, Michael Paquier, Euler Taveira Discussion: https://postgr.es/m/CAHut+Pt9_T6pWar0FLtPsygNmme8HPWPdGUyZ_8mE1Yvjdf0ZA@mail.gmail.com https://git.postgresql.org/pg/commitdiff/fc749bc7041cb77b5f6b58c129ad2616a3f7ab4f Ensure unlinking of old index file with REINDEX (TABLESPACE). The original versions of the patch included this part, but a mismerge from my side has made this piece go missing. Oversight in c5b28604. https://git.postgresql.org/pg/commitdiff/5128483d064038702f535aced2cbaa43256db214 Clarify some comments around SharedRecoveryState in xlog.c. SharedRecoveryState has been switched from a boolean to an enum as of commit 4e87c48, but some comments still referred to it as a boolean. Author: Amul Sul Reviewed-by: Dilip Kumar, Kyotaro Horiguchi Discussion: https://postgr.es/m/CAAJ_b97Hf+1SXnm8jySpO+Fhm+-VKFAAce1T_cupUYtnE3Nxig https://git.postgresql.org/pg/commitdiff/f7400823c3bd6ce03c2fe1bec5b7066bad146809
Peter Eisentraut pushed: SEARCH and CYCLE clauses. This adds the SQL standard feature that adds the SEARCH and CYCLE clauses to recursive queries to be able to do produce breadth- or depth-first search orders and detect cycles. These clauses can be rewritten into queries using existing syntax, and that is what this patch does in the rewriter. Reviewed-by: Vik Fearing vik@postgresfriends.org Reviewed-by: Pavel Stehule pavel.stehule@gmail.com Discussion: https://www.postgresql.org/message-id/flat/db80ceee-6f97-9b4a-8ee8-3ba0c58e5be2@2ndquadrant.com https://git.postgresql.org/pg/commitdiff/3696a600e2292d43c00949ddf0352e4ebb487e5b Improve confusing variable names. The prototype calls the second argument of pgstat_progress_update_multi_param() "index", and some callers name their local variable that way. But when the surrounding code deals with index relations, this is confusing, and in at least one case shadowed another variable that is referring to an index relation. Adjust those call sites to have clearer local variable naming, similar to existing callers in indexcmds.c. https://git.postgresql.org/pg/commitdiff/1d71f3c83c113849fe3aa60cb2d2c12729485e97 pg_dump: Fix dumping of inherited generated columns. Generation expressions of generated columns are always inherited, so there is no need to set them separately in child tables, and there is no syntax to do so either. The code previously used the code paths for the handling of default values, for which different rules apply; in particular it might want to set a default value explicitly for an inherited column. This resulted in unrestorable dumps. For generated columns, just skip them in inherited tables. Reviewed-by: Tom Lane tgl@sss.pgh.pa.us Discussion: https://www.postgresql.org/message-id/flat/15830.1575468847%40sss.pgh.pa.us https://git.postgresql.org/pg/commitdiff/0bf83648a52df96f7c8677edbbdf141bfa0cf32b Refactor Windows error message for easier translation. In the error messages referring to the user right "Lock pages in memory", this is a term from the Windows OS, so it should be translated in accordance with the OS localization. Refactor the error messages so this is easier and clearer. Also fix the capitalization to match the existing capitalization in the OS. https://git.postgresql.org/pg/commitdiff/3c78e0569ca04f4c92f0adcd74471398bb7b2e55
Robert Haas pushed: Factor pattern-construction logic out of processSQLNamePattern. The logic for converting the shell-glob-like syntax supported by utilities like psql and pg_dump to regular expression is extracted into a new function patternToSQLRegex. The existing function processSQLNamePattern now uses this function as a subroutine. patternToSQLRegex is a little more general than what is required by processSQLNamePattern. That function is only interested in patterns that can have up to 2 parts, a schema and a relation; but patternToSQLRegex can limit the maximum number of parts to between 1 and 3, so that patterns can look like either "database.schema.relation", "schema.relation", or "relation" depending on how it's invoked and what the user specifies. processSQLNamePattern only passes two buffers, so works exactly the same as before, always interpreting the pattern as either a "schema.relation" pattern or a "relation" pattern. But, future callers can use this function in other ways. Mark Dilger, reviewed by me. The larger patch series of which this is a part has also had review from Peter Geoghegan, Andres Freund, Álvaro Herrera, Michael Paquier, and Amul Sul, but I don't know whether any of them have reviewed this bit specifically. Discussion: http://postgr.es/m/12ED3DA8-25F0-4B68-937D-D907CFBF08E7@enterprisedb.com Discussion: http://postgr.es/m/5F743835-3399-419C-8324-2D424237E999@enterprisedb.com Discussion: http://postgr.es/m/70655DF3-33CE-4527-9A4D-DDEB582B6BA0@enterprisedb.com https://git.postgresql.org/pg/commitdiff/2c8726c4b0a496608919d1f78a5abc8c9b6e0868 Move some code from src/bin/scripts to src/fe_utils to permit reuse. The parallel slots infrastructure (which implements client-side multiplexing of server connections doing similar things, not threading or multiple processes or anything like that) are moved from src/bin/scripts/scripts_parallel.c to src/fe_utils/parallel_slot.c. The functions consumeQueryResult() and processQueryResult() which were previously part of src/bin/scripts/common.c are now moved into that file as well, becoming static helper functions. This might need to be changed in the future, but currently they're not used for anything else. Some other functions from src/bin/scripts/common.c are moved to to src/fe_utils and are split up among several files. connectDatabase(), connectMaintenanceDatabase(), and disconnectDatabase() are moved to connect_utils.c. executeQuery(), executeCommand(), and executeMaintenanceCommand() are move to query_utils.c. handle_help_version_opts() is moved to option_utils.c. Mark Dilger, reviewed by me. The larger patch series of which this is a part has also had review from Peter Geoghegan, Andres Freund, Álvaro Herrera, Michael Paquier, and Amul Sul, but I don't know whether any of them have reviewed this bit specifically. Discussion: http://postgr.es/m/12ED3DA8-25F0-4B68-937D-D907CFBF08E7@enterprisedb.com Discussion: http://postgr.es/m/5F743835-3399-419C-8324-2D424237E999@enterprisedb.com Discussion: http://postgr.es/m/70655DF3-33CE-4527-9A4D-DDEB582B6BA0@enterprisedb.com https://git.postgresql.org/pg/commitdiff/e955bd4b6c2bcdbd253837f6cf4c7520b98e69d4 Generalize parallel slot result handling. Instead of having a hard-coded behavior that we ignore missing tables and report all other errors, let the caller decide what to do by setting a callback. Mark Dilger, reviewed and somewhat revised by me. The larger patch series of which this is a part has also had review from Peter Geoghegan, Andres Freund, Álvaro Herrera, Michael Paquier, and Amul Sul, but I don't know whether any of them have reviewed this bit specifically. Discussion: http://postgr.es/m/12ED3DA8-25F0-4B68-937D-D907CFBF08E7@enterprisedb.com Discussion: http://postgr.es/m/5F743835-3399-419C-8324-2D424237E999@enterprisedb.com Discussion: http://postgr.es/m/70655DF3-33CE-4527-9A4D-DDEB582B6BA0@enterprisedb.com https://git.postgresql.org/pg/commitdiff/418611c84d004f45d92bcaa3f8e100385d96cd41
Heikki Linnakangas pushed: Fix small error in COPY FROM progress reporting. The # of bytes processed was accumulated slightly incorrectly. After loading more data to the input buffer, we added the number of bytes in the buffer to the sum. But in case of multi-byte characters or escapes, there can be a few unprocessed bytes left over from previous load in the buffer. Those bytes got counted twice. https://git.postgresql.org/pg/commitdiff/2f86ab305e7fbc7b84960079551cf9cafd29684f Fix backslash-escaping multibyte chars in COPY FROM. If a multi-byte character is escaped with a backslash in TEXT mode input, and the encoding is one of the client-only encodings where the bytes after the first one can have an ASCII byte "embedded" in the char, we didn't skip the character correctly. After a backslash, we only skipped the first byte of the next character, so if it was a multi-byte character, we would try to process its second byte as if it was a separate character. If it was one of the characters with special meaning, like '\n', '\r', or another '\', that would cause trouble. One such exmple is the byte sequence '\x5ca45c2e666f6f' in Big5 encoding. That's supposed to be [backslash][two-byte character][.][f][o][o], but because the second byte of the two-byte character is 0x5c, we incorrectly treat it as another backslash. And because the next character is a dot, we parse it as end-of-copy marker, and throw an "end-of-copy marker corrupt" error. Backpatch to all supported versions. Reviewed-by: John Naylor, Kyotaro Horiguchi Discussion: https://www.postgresql.org/message-id/a897f84f-8dca-8798-3139-07da5bb38728%40iki.fi https://git.postgresql.org/pg/commitdiff/c444472af5c202067a9ecb0ff8df7370fb1ea8f4
Peter Geoghegan pushed: Harden nbtree page deletion. Add some additional defensive checks in the second phase of index deletion to detect and report index corruption during VACUUM, and to avoid having VACUUM become stuck in more cases. The code is still not robust in the presence of a circular chain of sibling links, though it's not clear whether that really matters. This is follow-up work to commit 3a01f68e. The new defensive checks rely on the assumption that there can be no more than one VACUUM operation running for an index at any given time. Remove an old comment suggesting that multiple concurrent VACUUMs need to be considered here. This concern now seems highly unlikely to have any real validity, since we clearly rely on the same assumption in several other places. For example, there are much more recent comments that appear in the same function (added by commit efada2b8e92) that make the same assumption. Also add a CHECK_FOR_INTERRUPTS() to the relevant code path. Contrary to comments added by commit 3a01f68e, it is actually possible to handle interrupts here, at least in the common case where processing takes place at the leaf level. We only hold a pin on leafbuf/target page when stepping right at the leaf level. No backpatch due to the lack of complaints following hardening added to the same area by commit 3a01f68e. https://git.postgresql.org/pg/commitdiff/c34787f910585f82320f78b0afd53a6a170aa229 Rename removable xid function for consistency. GlobalVisIsRemovableFullXid() is now GlobalVisCheckRemovableFullXid(). This is consistent with the general convention for FullTransactionId equivalents of functions that deal with TransactionId values. It now matches the nearby GlobalVisCheckRemovableXid() function, which performs the same check for callers that use TransactionId values. Oversight in commit dc7420c2c92. Discussion: https://postgr.es/m/CAH2-Wzmes12jFNDcVgpU89Vp=r6uLFrE-MT0fjSWGsE70UiNaA@mail.gmail.com https://git.postgresql.org/pg/commitdiff/617fffee8a6f350ff03069e2843ecd039ea06ccc
Thomas Munro pushed: Etsuro Fujita pushed: - postgres_fdw: Fix assertion in estimate_path_cost_size(). Commit 08d2d58a2 added an assertion assuming that the retrieved_rows estimate for a foreign relation, which is re-used to cost pre-sorted foreign paths with local stats, is set to at least one row in estimate_path_cost_size(), which isn't correct because if the relation is a foreign table with tuples=0, the estimate would be set to 0 there when not using remote estimates. Per bug #16807 from Alexander Lakhin. Back-patch to v13 where the aforementioned commit went in. Author: Etsuro Fujita Reviewed-by: Kyotaro Horiguchi Discussion: https://postgr.es/m/16807-9fe4e08fbaa5c7ce%40postgresql.org https://git.postgresql.org/pg/commitdiff/5e7fa189ee92d5ecf42a295c336625d71bfe876d
Tatsuo Ishii pushed: Pending PatchesKyotaro HORIGUCHI sent in another revision of a patch to make it possible to use a directory of CRLs (certificate revocation lists) per the X.509 spec. Zeng Wenjing sent in another revision of a patch to implement global temporary tables. Scott Mead sent in a patch to allow the cost_limit to be re-calculated up to the maximum allowable (currently 10,000). This has the effect of allowing users to reload a configuration change and an in-progress vacuum can be ‘sped-up’ by setting either the cost_limit or cost_delay Paul Martinez sent in another revision of a patch to clarify whether a logical or physical replication connection was being rejected by pg_hba.conf rules. Thomas Munro sent in another revision of a patch to use a global barrier to fix DROP TABLESPACE on Windows, and use condition variables for ProcSignalBarriers. Euler Taveira de Oliveira sent in another revision of a patch to implement row filtering for logical replication. Aleksey Kondratov and Michaël Paquier traded patches to enable CLUSTER, VACUUM FULL and REINDEX to change tablespace on the fly. Etsuro Fujita sent in another revision of a patch to implement asynchronous Append on postgres_fdw nodes. Hou Zhijie and Greg Nancarrow traded patches to add a GUC and capability to run DML in parallel. Daniel Gustafsson and Jacob Champion traded patches to make it possible to use NSS as a TLS backend for libpq. Heikki Linnakangas sent in two more revisions of a patch to perform COPY FROM encoding conversions in larger chunks. Bruce Momjian sent in another revision of a patch to implement key management. John Naylor sent in two revisions of a patch to verify UTF-8 using SIMD instructions. Amit Kapila, Peter Smith, and Takamichi Osumi traded patches to enable enable tablesync workers to run in parallel. Pavel Stěhule sent in another revision of a patch to implement schema variables. Justin Pryzby sent in a patch to remove deprecated v8.2 containment operators. Noah Misch sent in a patch to fix a race between KeepFileRestoredFromArchive() and restartpoint. Peter Eisentraut sent in a patch to improve the new hash partition bound check error messages by adding a detail message that shows the particular numbers involved. Iwata Aya sent in another revision of a patch to add tracing capability to libpq. Julien Rouhaud sent in three revisions of a patch to allow HEAP_XMAX_LOCK_ONLY and HEAP_KEYS_UPDATED combination. This combination of hint bits was previously detected as a form of corruption, but it can be obtained with some mix of SELECT ... FOR UPDATE and UPDATE queries Atsushi Torikoshi and Fujii Masao traded patches to add a wait_start column to pg_locks. Greg Nancarrow sent in two more revisions of a patch to implement INSERT ... SELECT in parallel. Mark Rofail sent in four more revisions of a patch to implement foreign key arrays. Álvaro Herrera sent in another revision of a patch to add pg_atomic_monotonic_advance_u64, and use same to make LogwrtResult atomic. David Rowley sent in another revision of a patch to implement a Result Cache node and use same to cache results from subplans. Vigneshwaran C sent in another revision of a patch to make it possible to print a backtrace of a specified postgres process using a new pg_print_backtrace() function accessible to database superusers. Peter Eisentraut sent in a patch to pg_dump to add const decorations to the info arguments of the dump functions, in order to clarify that they don't modify that argument. Daniel Gustafsson sent in another revision of a patch to support checksum enable/disable in a running instance. Peter Smith sent in a patch intended to fix a bug that manifested as DROP TABLE breaks sync worker relid. Heikki Linnakangas sent in two more revisions of a patch to remove server and libpq support for old FE/BE protocol version 2, and simplify COPY FROM parsing by forcing lookahead. Mark Dilger sent in two more revisions of a patch to implement pg_amcheck. Tomáš Vondra sent in another revision of a patch to implement BRIN multi-range indexes. Bharath Rupireddy sent in another revision of a patch to postgres_fdw which adds keep_connections GUCs at both the FDW and at the global level that tells to not cache connections. David Rowley sent in another revision of a patch to implement tid scans, as distinct from the existing tid probes. Bertrand Drouvot sent in another revision of a patch to implement minimal logical decoding on standbys. Bruce Momjian sent in two revisions of a patch intended to fix a bug that manifested as multiple full page writes in a single checkpoint. Amit Langote sent in another revision of a patch to make updates and deletes on inheritance trees scale better. Ronan Dunklau and Michaël Paquier traded patches to preserve attstattarget on REINDEX CONCURRENTLY. Shenhao Wang, Kyotaro HORIGUCHI, and Hayato Kuroda traded patches to fix a parsing mistake in ecpg connect strings. Dilip Kumar sent in three more revisions of a patch to provide a new interface to get the recovery pause status. Peter Smith and Amit Kapila traded patches to make pg_replication_origin_drop safe against concurrent drops. Amit Langote sent in another revision of a patch to prevent FDW insert batching during cross-partition updates. Li Japin sent in another revision of a patch to implement ALTER SUBSCRIPTION ... ADD/DROP PUBLICATION. Stephen Frost sent in another revision of a patch to improve logging of auto-vacuum and auto-analyze. Jacob Champion sent in a patch to tweak the Kerberos tests. Dilip Kumar sent in two more revisions of a patch to implement custom compression methods for tables. Masahiro Ikeda sent in another revision of a patch to add WAL write/fsync statistics to pg_stat_wal. Justin Pryzby sent in another revision of a patch to make CLUSTER work on partitioned indexes. Heikki Linnakangas sent in a patch to get psql's \copy to send data to the server in larger chunks. Kazutaka Onishi sent in two more revisions of a patch to implement TRUNCATE on foreign tables generally and in the PostgreSQL FDW specifically. Tom Lane sent in another revision of a patch to fix postgres_fdw collation handling. Pavel Stěhule sent in another revision of a patch to enhance the PL/pgsql debug API by returning the text value of variable content. Haiying Tang sent in a patch to support tab completion for upper case character inputs in psql. Atsushi Torikoshi sent in another revision of a patch to add plan type to pg_stat_statements. Takamichi Osumi sent in two more revisions of a patch to add tests to for the tablesync worker. Michaël Paquier sent in another revision of a patch to add a PROCESS_TOAST option to VACUUM. |