Thread: BUG #5959: pg_restore --use-list does not always work with a pg_restore generated TOC file

The following bug has been logged online:

Bug reference:      5959
Logged by:          Per-Olov Esgard
Email address:      Per-Olov.Esgard@micronic-mydata.com
PostgreSQL version: 9.0.0
Operating system:   Linux / kernel 2.6.16.20
Description:        pg_restore --use-list  does not always work with a
pg_restore generated TOC file
Details:

General
=======
In my application I am using automatically generated psm:s. These psm: are
using many arguments, in some cases over 80. This means that a TOC entry in
a TOC file generated by pg_restore --list may be very long for psm:s with
such a long signature.

Specific
========
My backup system makes a schema dump to a file and a data dump to another
file. When performing a restore I create a TOC list from the schema dump by
using pg_restore on the schema dump file. I do some filtering on the TOC
file and then I restore it using pg_restore --use-list. This does not work
since some entries in the TOC file are more than 2000 characters long. The
function SortTocFromFile in pg_backup_archiver.c has a hard coded limitation
of 1024 characters for each row in the TOC file.

Temporary solution
==================
I have now solved this problem in my application by extending the buffer to
4096 characters and recompiled pg_restore.

"Per-Olov Esgard" <Per-Olov.Esgard@micronic-mydata.com> writes:
> Description:        pg_restore --use-list  does not always work with a
> pg_restore generated TOC file
> Details:

> General
> =======
> In my application I am using automatically generated psm:s. These psm: are
> using many arguments, in some cases over 80. This means that a TOC entry in
> a TOC file generated by pg_restore --list may be very long for psm:s with
> such a long signature.

> Specific
> ========
> My backup system makes a schema dump to a file and a data dump to another
> file. When performing a restore I create a TOC list from the schema dump by
> using pg_restore on the schema dump file. I do some filtering on the TOC
> file and then I restore it using pg_restore --use-list. This does not work
> since some entries in the TOC file are more than 2000 characters long. The
> function SortTocFromFile in pg_backup_archiver.c has a hard coded limitation
> of 1024 characters for each row in the TOC file.

Hm.  The function only cares about the dump ID at the start of the line.
AFAICS the consequence of buffer overflow would be that it'd take a line
continuation as a new line; which would generally result in a harmless
"WARNING: line ignored" message.  You didn't say exactly what symptom
you were seeing, but "does not work" seems like a bit of an
overstatement for that.  So I'm wondering whether you're seeing some
behavior I'm missing.

            regards, tom lane

I wrote:
> Per-Olov Esgard <Per-Olov.Esgard@micronic-mydata.com> writes:
>> The third extracted buffer is not ok but interpreted as ok with id 10:
>> 10."boolean", mydbcomndata_10."boolean", mydbcomndata_10."boolean",
>> mydbcomndata_10."boolean", mydbcomndata_10."boolean", character varying,
>> mydbcomndata_10."integer", mydbcomndata_10.componentname,
>> mydbcomndata_10."integer", mydbcomndata_10."integer",
>> mydbcomndata_10."integer", mydbcomndata_10.angle,
>> mydbcomndata_10."boolean", mydbcomndata_10."boolean",
>> mydbcomndata_10."boolean", mydbcomndata_10."boolean",
>> mydbcomndata_10."boolean", mydbcomndata_10."boolean",
>> mydbcomndata_10."boolean") postgres

>> So as you can see, the last part of  my schema name consists of a number
>> that is interpreted as an id.

> Ah, I see.  So what we probably need to do for a real fix is to teach
> that code to distinguish continuation lines from normal ones.

Fix committed --- thanks for the report!

            regards, tom lane