Thread: #XX000: ERROR: tuple concurrently updated
Hi. A tester just tried to restore two custom backups (not official PostgreSQL ones) concurrently.
The restore process creates a new schema, some roles,and does some role-to-role grants, before loading the data.
The two restores are independent, i.e. create different non-overlapping schemas.
But both are "registered" in the same pre-existing schema, SCH1.
And both run within a single transaction, that does everything, DDLs and DMLs.
The first session was doing a longer bigger restore, and was started first.
While the second session was doing a faster shorter one, and started a few seconds later, concurrently.
The second session completed OK.
But the first session errors out with:
Error: DDL Error: GRANT USAGE ON SCHEMA "SCH1", "SCH2" TO "SCH2:RO", "SCH2:RW", "SCH2:SU": #XX000: ERROR: tuple concurrently updated
Thus I'm trying to understand what's going on.
As noted above, SCH1 is the pre-existing schema, while SCH2 is the new one created by the restore.
The second session that completes normally did a `GRANT USAGE ON SCHEMA "SCH1", "SCH3" TO "SCH3:RO", "SCH3:RW", "SCH3:SU"` within its own transaction.
When I research #XX000, it often mentions doing a SELECT FOR UPDATE, but here I'm doing GRANTs, so I don't think that applies.
Is the issue related to trying to change SCHEMA ACLs for SCH1 concurrently, in two long running transactions? How am I supposed to resolve this?
I have to confess being a little surprised and blindsided by this issue.
Help understanding what's going on, and how to fix it, would be greatly appreciated.
Thanks, --DD
Dominique Devienne <ddevienne@gmail.com> writes:
> Hi. A tester just tried to restore two custom backups (not official
> PostgreSQL ones) concurrently.
> ...
> The second session completed OK.
> But the first session errors out with:
> Error: DDL Error: GRANT USAGE ON SCHEMA "SCH1", "SCH2" TO "SCH2:RO",
> "SCH2:RW", "SCH2:SU": #XX000: ERROR: tuple concurrently updated
> Thus I'm trying to understand what's going on.
Since both restores tried to grant some permissions on SCH1, they
both had to update SCH1's pg_namespace row (specifically nspacl).
We have no support for concurrent updates in the catalog-manipulation
code, so if the second run arrives at that step before the first
one has committed its pg_namespace change, you get this error.
> Is the issue related to trying to change SCHEMA ACLs for SCH1 concurrently,
> in two long running transactions? How am I supposed to resolve this?
The window is probably too small to hit if each restore is committing
as it goes, but if you run in --single-transaction mode then this
isn't surprising. I'd say don't try to run concurrent restores.
regards, tom lane
On Thu, Feb 20, 2025 at 4:27 PM Tom Lane <tgl@sss.pgh.pa.us> wrote:
Dominique Devienne <ddevienne@gmail.com> writes:
> Hi. A tester just tried to restore two custom backups (not official
> PostgreSQL ones) concurrently.
> ...
> The second session completed OK.
> But the first session errors out with:
> Error: DDL Error: GRANT USAGE ON SCHEMA "SCH1", "SCH2" TO "SCH2:RO",
> "SCH2:RW", "SCH2:SU": #XX000: ERROR: tuple concurrently updated
> Thus I'm trying to understand what's going on.
Since both restores tried to grant some permissions on SCH1, they
both had to update SCH1's pg_namespace row (specifically nspacl).
We have no support for concurrent updates in the catalog-manipulation
code, so if the second run arrives at that step before the first
one has committed its pg_namespace change, you get this error.
Thanks for confirming Tom.
> Is the issue related to trying to change SCHEMA ACLs for SCH1 concurrently,
> in two long running transactions? How am I supposed to resolve this?
The window is probably too small to hit if each restore is committing
as it goes, but if you run in --single-transaction mode then this
isn't surprising. I'd say don't try to run concurrent restores.
First, I'm not in psql, but my own code. And as mentioned, I'm already in the equivalent,
doing everything (DDLs and DMLs) in a single transaction.
Second, not doing concurrent restores is not an option.
So I need to separate operations related to SCH2,3,..., which are all independent,
and SCH1 which is shared, as separate long running and short transactions, respectively.
With perhaps some retry logic on the SCH1, just in case. --DD
Since you are willing to break the all one transaction rule, and if the restores were created via pg_dump, you could use the --section argument to split things up, run the "pre-data" sections serially, and the rest ("data" and "post-data") concurrently.
Cheers,
Greg
--
Crunchy Data - https://www.crunchydata.com
Enterprise Postgres Software Products & Tech Support
On Thu, Feb 20, 2025 at 4:27 PM Tom Lane <tgl@sss.pgh.pa.us> wrote:
Dominique Devienne <ddevienne@gmail.com> writes:
> Error: DDL Error: GRANT USAGE ON SCHEMA "SCH1", "SCH2" TO "SCH2:RO",
> "SCH2:RW", "SCH2:SU": #XX000: ERROR: tuple concurrently updated
Since both restores tried to grant some permissions on SCH1, they
both had to update SCH1's pg_namespace row (specifically nspacl).
We have no support for concurrent updates in the catalog-manipulation
code, so if the second run arrives at that step before the first
one has committed its pg_namespace change, you get this error.
Hi Tom, and al.
I have a related question, on role-to-role grants this time.
Above, it was contention on pg_namespace.nspacl in two transactions.
But during those "restore" transactions, I must also make role-to-role grants,
which AFAIK involve adding rows to pg_auth_members. So they are not subject
to the same "no support for concurrent updates in the catalog-manipulation"
you mentioned, as schema-to-role grants are, right? Because that's an insert,
not an update? Just want to make sure, as I'm thinking how to change our code.
Thanks, --DD
Dominique Devienne <ddevienne@gmail.com> writes:
> But during those "restore" transactions, I must also make role-to-role
> grants,
> which AFAIK involve adding rows to pg_auth_members. So they are not subject
> to the same "no support for concurrent updates in the catalog-manipulation"
> you mentioned, as schema-to-role grants are, right? Because that's an
> insert,
> not an update? Just want to make sure, as I'm thinking how to change our
> code.
Yeah, I think "GRANT role" should be relatively immune to that
problem, as long as you're making distinct grants (not same
grantor/grantee/granted roles). Wouldn't hurt to test.
regards, tom lane