Preventing deadlock on parallel backup - Mailing list pgsql-hackers

From Lucas
Subject Preventing deadlock on parallel backup
Date
Msg-id CAEWGB6_tbkq6UpmHBjZu6rTEC+p0OLzCuWn-SeOv_ENizadx9Q@mail.gmail.com
Whole thread Raw
Responses Re: Preventing deadlock on parallel backup  (Tom Lane <tgl@sss.pgh.pa.us>)
List pgsql-hackers
<p dir="ltr">People,<p dir="ltr">I made a small modification in pg_dump to prevent parallel backup failures due to
exclusivelock requests made by other tasks. <p dir="ltr">The modification I made take shared locks for each parallel
backupworker at the very beginning of the job. That way, any other job that attempts to acquire exclusive locks will
waitfor the backup to finish.<p dir="ltr">In my case, each server was taking a day to complete the backup, now with
parallelbackup one is taking 3 hours and the others less than a hour.<p dir="ltr">The code below is not very elegant,
butit works for me. My whishlist for the backup is:<p dir="ltr">1) replace plpgsql by c code reading the backup toc and
assemblingthe lock commands.<br /> 2) create an timeout to the locks.<br /> 3) broadcast the end of copy to every
workerin order to release the locks as early as possible;<br /> 4) create a monitor thread that prioritize an copy job
basedon a exclusive lock acquired;<br /> 5) grant the lock for other connection of the same distributed transaction if
itis held by any connection of the same distributed transaction. There is some sideefect I can't see on that?<p
dir="ltr">1to 4 are within my capabilities and I may do it in the future. 4 is to advanced for me and I do not dare to
messwith something so fundamental rights now.<p dir="ltr">Anyone else is working on that?<br /><p dir="ltr">On,
Parallel.c,void RunWorker(...), add:<p dir="ltr">PQExpBuffer query;<br /> PGresult   *res;<p dir="ltr">query =
createPQExpBuffer();           <br /> resetPQExpBuffer(query);<br /> appendPQExpBuffer(query,<br /> "do language
'plpgsql'$$"<br /> " declare "<br /> "    x record;"<br /> " begin"<br /> "    for x in select * from pg_tables where
schemanamenot in ('pg_catalog','information_schema') loop"<br /> "        raise info 'lock table %.%', x.schemaname,
x.tablename;"<br/> "        execute 'LOCK TABLE '||quote_ident(x.schemaname)||'.'||quote_ident(x.tablename)||' IN
ACCESSSHARE MODE NOWAIT';"<br /> "    end loop;"<br /> "end"<br /> "$$" );<p dir="ltr">res = PQexec(AH->connection,
query->data);<pdir="ltr">if (!res || PQresultStatus(res) != PGRES_COMMAND_OK)<br />        
exit_horribly(modulename,"Couldnot lock the tables to begin the work\n\n");<br /> PQclear(res);<br />
destroyPQExpBuffer(query);<br/> 

pgsql-hackers by date:

Previous
From: Stephen Frost
Date:
Subject: Re: Add support for restrictive RLS policies
Next
From: Tom Lane
Date:
Subject: Re: Re: GiST optimizing memmoves in gistplacetopage for fixed-size updates [PoC]