Thread: pg_dump failure in tar format.
I've posted the information below twice to the admin list without getting a
solution, so I'm promoting it to a bug.
I'm not subscribed to the bug list, so please cc me on responses and I'll
try to supply information as needed.
I'm getting the following error message:
pg_dump: [tar archiver] could not write to tar member (wrote 39, attempted
166)
Here are the particulars:
-I'm running this command: "pg_dump -Ft prod > prod.dump.tar" (The database
is named prod)
-The dump gets about 1/4 of the way through, and then gives me the error
message and dies.
-I'm running PostgreSQL version 7.3.2.
-There is plenty of disk space available.
-The same command on the same database and server with same specs worked
last week when I was on V7.2.1.
-Since upgrading, more data has been added, but the structure of the
database is unchanged.
-Using the -v switch shows me that it always quits on the same table, but
otherwise adds no new information.
-The part of the error message in parentheses changes on each run. For
instance, on the last run, I got "(wrote 64, attempted 174)" The rest of
the message remains consistent.
-The table it quits on is fairly large- about 2.6GB. It is both "wide"
because it contains a text field that is usually a few sentences of text,
and "long", containing 9,137,808 records. This is also the only table in our
database that is split into multiple files.
-A text dump using this command works fine and exports the entire database
without a problem: "pg_dump prod > prod.dump.text"
-I have set up an identical system (Same hardware, same software, same Data
in DB) to do testing on, and confirmed that I get the same error on this
system, so it appears to not be a hardware error or just a bad copy of the
software.
-Several folks suggested that I was hitting the 2GB file size limit. This is
not the case. Here is a snip from the console log on the second machine that
I'm using to diagnose with:
nickf@morgai:~$ pg_dump -Ft alpha > dump.tar
pg_dump: [tar archiver] could not write to tar member (wrote 110, attempted
398)
nickf@morgai:~$
nickf@morgai:~$
nickf@morgai:~$ ls -al dump.tar
-rw-r--r-- 1 nickf nickf 1388367872 Jul 21 14:49 dump.tar
nickf@morgai:~$
Thanks for looking this over- please let me know if anyone has any ideas
about this.
-Regards,
-Nick
---------------------------------------------------------------------
Nick Fankhauser
nickf@doxpop.com Phone 1.765.965.7363 Fax 1.765.962.9788
doxpop - Court records at your fingertips - http://www.doxpop.com/
At 02:47 PM 1/08/2003 -0500, Nick Fankhauser - Doxpop wrote:
>pg_dump: [tar archiver] could not write to tar member (wrote 39, attempted
>166)
One of the nasty features of TAR format is that it needs to know the file
size before adding it to the archive. As a result, pg_dump stores the file
in the /tmp directory before moving it to the actual output file. For huge
files, this means /tmp must be able to cope with the uncompressed size of
the largest table. It's horrible, I know, which is why I use -Fc, but I'd
guess this is the cause of your error.
It uses tmpfile() to get a temp file, so I can't see a simple way to test
this, unless you can free up 2+GB in /tmp?
Please let me know if this is the cause, and if you can not test it, I will
try to send a patch to (temporarily) avoid using tmpfile(). Ideally, I
suppose pg_dump should support the ability to override the tmpfile() location.
Bye for now,
Philip
----------------------------------------------------------------
Philip Warner | __---_____
Albatross Consulting Pty. Ltd. |----/ - \
(A.B.N. 75 008 659 498) | /(@) ______---_
Tel: (+61) 0500 83 82 81 | _________ \
Fax: (+61) 03 5330 3172 | ___________ |
Http://www.rhyme.com.au | / \|
| --________--
PGP key available upon request, | /
and from pgp5.ai.mit.edu:11371 |/
Phillip- Thanks for the explanation of how this works- I'll check it out & let you know if that is the source of the problem in my case. If it is, perhaps I can submit a short paragraph for the documentation that will clue people in if they run into this in the future. I suppose it could be handled more gracefully in the code by specifying a tempfile location as you suggest, but since it isn't really broken, I'd vote that documenting the potential constraint is a valid and time-saving "fix" that will let you put off messing with the code until you have a more compelling reason to work with it. -NF > -----Original Message----- > From: Philip Warner [mailto:pjw@rhyme.com.au] > Sent: Friday, August 01, 2003 6:37 PM > To: nickf@ontko.com; pgsql-bugs@postgresql.org > Subject: Re: [BUGS] pg_dump failure in tar format. > > > At 02:47 PM 1/08/2003 -0500, Nick Fankhauser - Doxpop wrote: > >pg_dump: [tar archiver] could not write to tar member (wrote 39, > attempted > >166) > > One of the nasty features of TAR format is that it needs to know the file > size before adding it to the archive. As a result, pg_dump stores > the file > in the /tmp directory before moving it to the actual output file. > For huge > files, this means /tmp must be able to cope with the uncompressed size of > the largest table. It's horrible, I know, which is why I use -Fc, but I'd > guess this is the cause of your error. > > It uses tmpfile() to get a temp file, so I can't see a simple way to test > this, unless you can free up 2+GB in /tmp? > > Please let me know if this is the cause, and if you can not test > it, I will > try to send a patch to (temporarily) avoid using tmpfile(). Ideally, I > suppose pg_dump should support the ability to override the > tmpfile() location. > > Bye for now, > > Philip > > > > > ---------------------------------------------------------------- > Philip Warner | __---_____ > Albatross Consulting Pty. Ltd. |----/ - \ > (A.B.N. 75 008 659 498) | /(@) ______---_ > Tel: (+61) 0500 83 82 81 | _________ \ > Fax: (+61) 03 5330 3172 | ___________ | > Http://www.rhyme.com.au | / \| > | --________-- > PGP key available upon request, | / > and from pgp5.ai.mit.edu:11371 |/ >