Thread: ftell mismatch with expected position
I am getting the following warning when our program runs pg_dump.exe and the output is in custom format and send to standard out which is connected to a pipe (Windows platform).
pg_dump: [custom archiver] WARNING: ftell mismatch with expected position -- ftell used
The output of pg_dump is received by another process that copies it (and some other data) into a zip file. When I extract the db backup from the zip the backup is fine and I can restore it.
I have both tried the pg_dump.exe from the 9.5 alpha and a stable 9.4 both generate this warning.
When I change the format to plain the warning goes away.
When I call pg_dump from the commandline and let it send to standard out and redirect it to a file the warning does not occur but it does occur when I redirect to nul.
Is there a way to prevent this warning? I would think pg_dump shouldn't call ftell on a pipe.
Regards, Eelke
On 10/29/2015 02:51 AM, Eelke Klein wrote: > I am getting the following warning when our program runs pg_dump.exe and > the output is in custom format and send to standard out which is > connected to a pipe (Windows platform). > > pg_dump: [custom archiver] WARNING: ftell mismatch with expected > position -- ftell used Can you show us the exact command, with arguments, that you are using? Also what is the size of the dump file you are creating? > > The output of pg_dump is received by another process that copies it (and > some other data) into a zip file. When I extract the db backup from the > zip the backup is fine and I can restore it. > > I have both tried the pg_dump.exe from the 9.5 alpha and a stable 9.4 > both generate this warning. > When I change the format to plain the warning goes away. > When I call pg_dump from the commandline and let it send to standard out > and redirect it to a file the warning does not occur but it does occur > when I redirect to nul. > > Is there a way to prevent this warning? I would think pg_dump shouldn't > call ftell on a pipe. > > Regards, Eelke -- Adrian Klaver adrian.klaver@aklaver.com
Normally we call this from within our windows program where a lot of code is involved for setting up the environment, and creating the pipes and redirecting stdout, stderr and stdin. However I believe it is the same problem that can be caused with the following commandline in cmd.exe but you will need to have Coreutils for windows installed to get the cat command.
pg_dump.exe -U postgres --format=c mydb | cat > backupfile
The size of the backup doesn't matter. As long as there is atleast one table in the database the warning occurs. When I use --verbose it looks like I get one warning for every table in the database while it is dumping the data in the tables.
Eelke Klein
2015-10-30 14:53 GMT+01:00 Adrian Klaver <adrian.klaver@aklaver.com>:
On 10/29/2015 02:51 AM, Eelke Klein wrote:I am getting the following warning when our program runs pg_dump.exe and
the output is in custom format and send to standard out which is
connected to a pipe (Windows platform).
pg_dump: [custom archiver] WARNING: ftell mismatch with expected
position -- ftell used
Can you show us the exact command, with arguments, that you are using?
Also what is the size of the dump file you are creating?--
The output of pg_dump is received by another process that copies it (and
some other data) into a zip file. When I extract the db backup from the
zip the backup is fine and I can restore it.
I have both tried the pg_dump.exe from the 9.5 alpha and a stable 9.4
both generate this warning.
When I change the format to plain the warning goes away.
When I call pg_dump from the commandline and let it send to standard out
and redirect it to a file the warning does not occur but it does occur
when I redirect to nul.
Is there a way to prevent this warning? I would think pg_dump shouldn't
call ftell on a pipe.
Regards, Eelke
Adrian Klaver
adrian.klaver@aklaver.com
Please refrain from top-posting. On 2 November 2015 at 10:48, Eelke Klein <eelke@bolt.nl> wrote: > Normally we call this from within our windows program where a lot of code is > involved for setting up the environment, and creating the pipes and > redirecting stdout, stderr and stdin. However I believe it is the same > problem that can be caused with the following commandline in cmd.exe but you > will need to have Coreutils for windows installed to get the cat command. > > pg_dump.exe -U postgres --format=c mydb | cat > backupfile Do you also get the warning if you split those commands up, and if so, in which one? pg_dump.exe -U postgres --format=c mydb > backupfile cat backupfile > someotherfile > The size of the backup doesn't matter. As long as there is atleast one table > in the database the warning occurs. When I use --verbose it looks like I get > one warning for every table in the database while it is dumping the data in > the tables. > > Eelke Klein > > > 2015-10-30 14:53 GMT+01:00 Adrian Klaver <adrian.klaver@aklaver.com>: >> >> On 10/29/2015 02:51 AM, Eelke Klein wrote: >>> >>> I am getting the following warning when our program runs pg_dump.exe and >>> the output is in custom format and send to standard out which is >>> connected to a pipe (Windows platform). >>> >>> pg_dump: [custom archiver] WARNING: ftell mismatch with expected >>> position -- ftell used >> >> >> Can you show us the exact command, with arguments, that you are using? >> >> Also what is the size of the dump file you are creating? >> >> >>> >>> The output of pg_dump is received by another process that copies it (and >>> some other data) into a zip file. When I extract the db backup from the >>> zip the backup is fine and I can restore it. >>> >>> I have both tried the pg_dump.exe from the 9.5 alpha and a stable 9.4 >>> both generate this warning. >>> When I change the format to plain the warning goes away. >>> When I call pg_dump from the commandline and let it send to standard out >>> and redirect it to a file the warning does not occur but it does occur >>> when I redirect to nul. >>> >>> Is there a way to prevent this warning? I would think pg_dump shouldn't >>> call ftell on a pipe. >>> >>> Regards, Eelke >> >> >> >> -- >> Adrian Klaver >> adrian.klaver@aklaver.com > > -- If you can't see the forest for the trees, Cut the trees and you'll see there is no forest.
Alban Hertroys <haramrae@gmail.com> writes:
> Please refrain from top-posting.
> On 2 November 2015 at 10:48, Eelke Klein <eelke@bolt.nl> wrote:
>> Normally we call this from within our windows program where a lot of code is
>> involved for setting up the environment, and creating the pipes and
>> redirecting stdout, stderr and stdin. However I believe it is the same
>> problem that can be caused with the following commandline in cmd.exe but you
>> will need to have Coreutils for windows installed to get the cat command.
>>
>> pg_dump.exe -U postgres --format=c mydb | cat > backupfile
> Do you also get the warning if you split those commands up, and if so,
> in which one?
Presumably it doesn't happen without the use of the pipe notation.
I suspect that what's happening is that stdout isn't getting put into
binary mode, so that Microsoft's CR/NL translation corrupts the data.
If that's true, though, the resulting backup file should be corrupt;
is it valid according to "pg_restore backupfile"?
regards, tom lane
Presumably it doesn't happen without the use of the pipe notation.
Indeed, it only happens when a pipe is involved.
I suspect that what's happening is that stdout isn't getting put into
binary mode, so that Microsoft's CR/NL translation corrupts the data.
If that's true, though, the resulting backup file should be corrupt;
is it valid according to "pg_restore backupfile"?
Yes pg_restore restores it without problems. However I did notice that it is larger then normal.
I have also tested what happens on freebsd when using a pipe and there are no warnings and backup is expected size.
I dug into the source code of pg_dump and found that the warning is given by _getFilePos(ArchiveHandle *AH, lclContext *ctx) in pg_backup_custom.c but it won't if ctx->hasSeek is set to false. Which is initialized by a call to checkSeek in pg_backup_archiver.c. Unfortunatly I'm not yet able to build and debug ton windows to verify if a correction to this function would help.
regards, Eelke