Thread: pg_dump --data-only: is dump ordered to keep foreign-key-relations loadable?

I migrated one database from 8.0 to 8.1

That I used to do add "without oids" to all tables.

First step so:

pg_dump --schema-only  -U user database

the file was edited, all tables to "withoud oids"; and reloaded in 8.1

After that I

pg_dump --data-only -U user database

and tried to reload the data. But it fails on foreign keys: depending tables are being dumped before the tables they depend on.

I solved it by manually dumping the relevant tables and reloading them,

Now I cannot find documentation
- if pg_dump is supposed to produce a "ordered dump" so that not doing is a bug and I need to present a showcase
- or if it is simply not implemented and an enhancement request;
- or if it is even on a theoretical basis impossible to derive the correct order. [circular foreign keys came to my mind]


Harald

--
GHUM Harald Massa
persuasion python postgresql
Harald Armin Massa
Reinsburgstraße 202b
70197 Stuttgart
0173/9409607

Harald Armin Massa wrote:
 > [snipped text]
> pg_dump --data-only -U user database
>
> and tried to reload the data. But it fails on foreign keys: depending tables
> are being dumped before the tables they depend on.
>
> I solved it by manually dumping the relevant tables and reloading them,
>
> Now I cannot find documentation
> - if pg_dump is supposed to produce a "ordered dump" so that not doing is a
> bug and I need to present a showcase
> - or if it is simply not implemented and an enhancement request;
> - or if it is even on a theoretical basis impossible to derive the correct
> order. [circular foreign keys came to my mind]

Hi

There are three possibilities to solve this
1) Use pg_dump (or pg_restore, if you are using custom-format dumps)
with the --disable-trigger option (check the man-page for the exact
syntax). This will disable all triggers, including those which check the
foreign-key contraints during the restore

2) If all your foreign keys are defined as "deferrable", you can
wrap the data-loading in a transaction, and do "set contraints all
deferred" before loading the data. This will defer the constraint-checks
until you issue commit.

3) If you have a schema and data-dump in seperate files, you could
manually split the schema dump into two files, one containing all
table definitions, the other containing the f-k definitions. You
can then first restore the schema, then your data, and finally your
foreign keys.

The ordering of the three options in terms of speed is 1 < 3 < 2, I
believe - but 2 and 3 give you additional security, because they check
the foreign keys during the import. 1) relies on the fact the the
dump doesn't containt foreign-key violations.

greetings, Florian Pflug

Harald Armin Massa wrote:
> I migrated one database from 8.0 to 8.1
>
> That I used to do add "without oids" to all tables.
>
> First step so:
>
> pg_dump --schema-only  -U user database
>
> the file was edited, all tables to "withoud oids"; and reloaded in 8.1
>
> After that I
>
> pg_dump --data-only -U user database
>
> and tried to reload the data. But it fails on foreign keys: depending
> tables are being dumped before the tables they depend on.
>
> I solved it by manually dumping the relevant tables and reloading them,
>
> Now I cannot find documentation
> - if pg_dump is supposed to produce a "ordered dump" so that not doing
> is a bug and I need to present a showcase
> - or if it is simply not implemented and an enhancement request;
> - or if it is even on a theoretical basis impossible to derive the
> correct order. [circular foreign keys came to my mind]

You can do this by creating a table of contents from your dump:
    pg_restore -l -a ... > toc_file ...

Edit this to re-order the contents such that they restore in a safe
order and then restore using the modified TOC:
    pg_restore -L toc_file ...

That works for me at least. It is a bit of a pain to have to do this.
Maybe there is an easier way?

Disabling triggers only affects triggers - not keys so doesn't help.

Hope that helps
Pete
--
Whitebeam Web Application Server
http://www.whitebeam.org
------

Florian,

thank you very much for those solutions; I shortened them, resend them for "increased google presence" :)

The ordering of the three options in terms of speed is 1 < 3 < 2, I
believe - but 2 and 3 give you additional security, because they check
the foreign keys during the import. 1) relies on the fact the the
dump doesn't containt foreign-key violations.

STILL ... to help on with the development:

- is pg_dump supposed to produce a "ordered dump" with --data-only?
(within documentation there is only a warning that --table will not dump the depending ones)
[than I have to create a extract to reproduce that "bug"]

- or if it is simply not implemented and an enhancement request;
(so I should file an enhancement-request and a "warning" -patch to the pg_dump docu, that pg_dump --dataonly MUST be used with --disable-trigger or similiar for backup-purposes)

- or if it is even on a theoretical basis impossible to derive the correct order. [circular foreign keys came to my mind] -> I also should submit a patch to documentation
--
GHUM Harald Massa
persuasion python postgresql
Harald Armin Massa
Reinsburgstraße 202b
70197 Stuttgart
0173/9409607

Harald Armin Massa wrote:
> - is pg_dump supposed to produce a "ordered dump" with --data-only?
> (within documentation there is only a warning that --table will not dump
> the depending ones)
> [than I have to create a extract to reproduce that "bug"]
>
> - or if it is simply not implemented and an enhancement request;
> (so I should file an enhancement-request and a "warning" -patch to the
> pg_dump docu, that pg_dump --dataonly MUST be used with
> --disable-trigger or similiar for backup-purposes)
>
> - or if it is even on a theoretical basis impossible to derive the
> correct order. [circular foreign keys came to my mind] -> I also should
> submit a patch to documentation

Even _if_ it's theoretically possible, it would probably be too slow,
i'd say. I'd be possible if you dependency-graph between your tables is
acyclic - then simpily reordering the dump-order would be sufficient.

But if you have some sort of cycle, you'd at least need to give up
table-wise restoration, and instead e.g. restore 5 rows from t1, then
2 rows from t2, now 3 rows from _t1_ _again_. And if a _record_ some
way depends on itself (So, the dependency graph between you _records_
even shows circles), then you'd sometimes need to insert a record, then
insert a record that depends on the first, and then update the first
record to now depend on the second. or something similar. And, with
not-null contraints in place, this might be impossible without violating
them :-(.

So, I'd say you should either create backups using --disable-triggers,
or use the custom-format - which let's you specify --disable-triggers,
and a few other options at restore, rather then at backup time.

greetings, Florian Pflug