Thread: Data corruption while taking backup of database

Hi Experts,

We took back up of a database using the following command
pg_dump -a
Please note that when we took the back the database was available for
use simultaneously from various clients.

In the back up file that was generated we found some data that was
violating foreign key constraints. And there fore we were not able to
restore database back with this back up file.

What may be the possible reasons?

Also please tell me, is it safe if we allow concurrent modification of
data through DMLs and the same time take the back up using pg_dump?

Regards
Pushker

The information contained in this electronic message and any attachments to this message are intended for the exclusive
useof the addressee(s) and may contain proprietary, confidential or privileged information. If you are not the intended
recipient,you should not disseminate, distribute or copy this e-mail. Please notify the sender immediately and destroy
theoriginal message all copies of this message and any attachments. 
WARNING: Computer viruses can be transmitted via email. The recipient should check this email and any attachments for
thepresence of viruses. The company accepts no liability for any damage caused by any virus transmitted by this email. 

Please do not print this email unless it is absolutely necessary.

Here is the exact command we use

pg_dump --disable-triggers -v -a -U postgres -p 5432 -h 127.0.0.1 -n pam
-f  <backupfilename>

on the postgres database version 8.1.11

Regards,
Pushker Chaubey

Pushker Chaubey wrote:
> Hi Experts,
>
> We took back up of a database using the following command
> pg_dump -a
> Please note that when we took the back the database was available for
> use simultaneously from various clients.
>
> In the back up file that was generated we found some data that was
> violating foreign key constraints. And there fore we were not able to
> restore database back with this back up file.
>
> What may be the possible reasons?
>
> Also please tell me, is it safe if we allow concurrent modification of
> data through DMLs and the same time take the back up using pg_dump?
>
> Regards
> Pushker
>


The information contained in this electronic message and any attachments to this message are intended for the exclusive
useof the addressee(s) and may contain proprietary, confidential or privileged information. If you are not the intended
recipient,you should not disseminate, distribute or copy this e-mail. Please notify the sender immediately and destroy
theoriginal message all copies of this message and any attachments. 
WARNING: Computer viruses can be transmitted via email. The recipient should check this email and any attachments for
thepresence of viruses. The company accepts no liability for any damage caused by any virus transmitted by this email. 

Please do not print this email unless it is absolutely necessary.

Pushker Chaubey wrote:
> We took back up of a database using the following command
> pg_dump -a
> Please note that when we took the back the database was available for
> use simultaneously from various clients.
>
> In the back up file that was generated we found some data that was
> violating foreign key constraints. And there fore we were not able to
> restore database back with this back up file.
>
> What may be the possible reasons?
>
> Also please tell me, is it safe if we allow concurrent modification of
> data through DMLs and the same time take the back up using pg_dump?

Yes, that is safe. pg_dump takes a coherent snapshot of the database as
it is when the backup starts; modifications during the backup don't show
up in the backup. But clearly something has corrupted your database
either before the backup or during it.

Are the rows that violate the foreign key constraint still there in the
original database? When was the last time you backed up? Can you check
if the foreign key violation present in the previous backup as well?

What does the schema of the tables involved look like? Can you post the
offending rows?

(this has nothing to do with JDBC, BTW. pgsql-general mailing list would
be more appropriate)

--
   Heikki Linnakangas
   EnterpriseDB   http://www.enterprisedb.com

Hi Heikki Linnakangas,

Thanks for your quick response.
We will look for other possible reasons as mentioned by you.

Thanks and regards,
Pushker Chaubey

Heikki Linnakangas wrote:
> Pushker Chaubey wrote:
>> We took back up of a database using the following command
>> pg_dump -a
>> Please note that when we took the back the database was available for
>> use simultaneously from various clients.
>>
>> In the back up file that was generated we found some data that was
>> violating foreign key constraints. And there fore we were not able to
>> restore database back with this back up file.
>>
>> What may be the possible reasons?
>>
>> Also please tell me, is it safe if we allow concurrent modification
>> of data through DMLs and the same time take the back up using pg_dump?
>
> Yes, that is safe. pg_dump takes a coherent snapshot of the database
> as it is when the backup starts; modifications during the backup don't
> show up in the backup. But clearly something has corrupted your
> database either before the backup or during it.
>
> Are the rows that violate the foreign key constraint still there in
> the original database? When was the last time you backed up? Can you
> check if the foreign key violation present in the previous backup as
> well?
>
> What does the schema of the tables involved look like? Can you post
> the offending rows?
>
> (this has nothing to do with JDBC, BTW. pgsql-general mailing list
> would be more appropriate)
>


The information contained in this electronic message and any attachments to this message are intended for the exclusive
useof the addressee(s) and may contain proprietary, confidential or privileged information. If you are not the intended
recipient,you should not disseminate, distribute or copy this e-mail. Please notify the sender immediately and destroy
theoriginal message all copies of this message and any attachments. 
WARNING: Computer viruses can be transmitted via email. The recipient should check this email and any attachments for
thepresence of viruses. The company accepts no liability for any damage caused by any virus transmitted by this email. 

Please do not print this email unless it is absolutely necessary.

Pushker Chaubey <pchaubey@vertex.co.in> writes:
> We took back up of a database using the following command
> pg_dump -a

> In the back up file that was generated we found some data that was
> violating foreign key constraints. And there fore we were not able to
> restore database back with this back up file.

Was the data actually erroneous, or just loaded in the wrong order
(ie, referencing table before referenced one)?  pg_dump -a doesn't
guarantee to use a safe reload order.

            regards, tom lane

Hi Tom,

We actually follow the following order while doing restore

1) disable triggers
2) drop the foreign key constraints only
3) truncate all the existing tables
4) do the restore
5) recreate the constraints
6) enable triggers

Regards,
Pushker Chaubey


Tom Lane wrote:

Pushker Chaubey <pchaubey@vertex.co.in> writes: 

We took back up of a database using the following command
pg_dump -a   

In the back up file that was generated we found some data that was 
violating foreign key constraints. And there fore we were not able to 
restore database back with this back up file.   

Was the data actually erroneous, or just loaded in the wrong order
(ie, referencing table before referenced one)?  pg_dump -a doesn't
guarantee to use a safe reload order.
		regards, tom lane
 

The information contained in this electronic message and any attachments to this message are intended for the exclusive use of the addressee(s) and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you should not disseminate, distribute or copy this e-mail. Please notify the sender immediately and destroy the original message all copies of this message and any attachments. WARNING: Computer viruses can be transmitted via email. The recipient should check this email and any attachments for the presence of viruses. The company accepts no liability for any damage caused by any virus transmitted by this email. Please do not print this email unless it is absolutely necessary.

Hello Heikki,

the problem is just the order of the exported tables. We always use
begin and commit surrounding the dump when restoring, and ensure, all
contraints are DEFERRABLE INITIALLY DEFERRED (You can alternatively SET
constraints ALL DEFERRED; after creating the schema and before entering
the data).

With best regards,
Daniel Migowski

Heikki Linnakangas schrieb:
> Pushker Chaubey wrote:
>> We took back up of a database using the following command
>> pg_dump -a
>> Please note that when we took the back the database was available for
>> use simultaneously from various clients.
>>
>> In the back up file that was generated we found some data that was
>> violating foreign key constraints. And there fore we were not able to
>> restore database back with this back up file.
>>
>> What may be the possible reasons?
>>
>> Also please tell me, is it safe if we allow concurrent modification
>> of data through DMLs and the same time take the back up using pg_dump?
>
> Yes, that is safe. pg_dump takes a coherent snapshot of the database
> as it is when the backup starts; modifications during the backup don't
> show up in the backup. But clearly something has corrupted your
> database either before the backup or during it.
>
> Are the rows that violate the foreign key constraint still there in
> the original database? When was the last time you backed up? Can you
> check if the foreign key violation present in the previous backup as
> well?
>
> What does the schema of the tables involved look like? Can you post
> the offending rows?
>
> (this has nothing to do with JDBC, BTW. pgsql-general mailing list
> would be more appropriate)
>