Thread: BUG #15333: pg_dump error on large table -- "pg_dump: could not statfile...iso-8859-1 error"

The following bug has been logged on the website:

Bug reference:      15333
Logged by:          Mark Lai
Email address:      mark.lai@integrafec.com
PostgreSQL version: 10.4
Operating system:   Windows Server 2016 Server
Description:

Hi,

When I pg_dump a large table (> 35 GB), I get the following error message:


pg_dump: could not stat file
"O:\postgres-server3\test#test#large_test/2793.dat.gz": Unknown error

The dump however appears to restore correctly.

**************************************
**Script used to create the table**
**************************************
CREATE SCHEMA IF NOT EXISTS test;

DROP TABLE IF EXISTS test.large_test;
CREATE UNLOGGED TABLE test.large_test (num1 bigint, num2 double precision,
num3 double precision);

INSERT INTO test.large_test (num1, num2, num3)
  SELECT round(random()*10), random(), random()*142
  FROM generate_series(1, 40*20000000) s(i);
  --20000000 ~ 1GB

*****************************************************
**power shell script used to launch pg_dump**
*****************************************************
$database = "test"
$schema = "test"
$table = "large_test"

$baseOutputDirectory = "O:\postgres-server3\"
$baseLogDirectory = "O:\postgres-server3\logs\"

############################


New-Item -ItemType Directory -Force -Path $baseOutputDirectory 
New-Item -ItemType Directory -Force -Path $baseLogDirectory


$schemaTable = "$($database)#$($schema)#$($table)"
$outputDirectory = "$($baseOutputDirectory)$($schemaTable)"
$logFile = "$($baseLogDirectory)$($schemaTable).log"

Remove-Item $outputDirectory -Force -Recurse -ErrorAction Ignore
Remove-Item $logFile -Force -Recurse -ErrorAction Ignore

Start-Job -Name $schemaTable -ScriptBlock{& "C:\Program
Files\PostgreSQL\10\bin\pg_dump.exe" --verbose --host=localhost --port=5432
--username=test123123 --no-password --jobs=1 --format=directory
--table=$Using:schema.$Using:table --file=$Using:outputDirectory
$Using:database 1> $Using:logFile 2>&1 }

************
**Log file**
*************
pg_dump.exe : pg_dump: last built-in OID is 16383
At line:1 char:1
+ & "C:\Program Files\PostgreSQL\10\bin\pg_dump.exe" --verbose --host=l
...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : NotSpecified: (pg_dump: last built-in OID is
16383:String) [], RemoteException
    + FullyQualifiedErrorId : NativeCommandError
 
pg_dump: reading extensions
pg_dump: identifying extension members
pg_dump: reading schemas
pg_dump: reading user-defined tables
pg_dump: reading user-defined functions
pg_dump: reading user-defined types
pg_dump: reading procedural languages
pg_dump: reading user-defined aggregate functions
pg_dump: reading user-defined operators
pg_dump: reading user-defined access methods
pg_dump: reading user-defined operator classes
pg_dump: reading user-defined operator families
pg_dump: reading user-defined text search parsers
pg_dump: reading user-defined text search templates
pg_dump: reading user-defined text search dictionaries
pg_dump: reading user-defined text search configurations
pg_dump: reading user-defined foreign-data wrappers
pg_dump: reading user-defined foreign servers
pg_dump: reading default privileges
pg_dump: reading user-defined collations
pg_dump: reading user-defined conversions
pg_dump: reading type casts
pg_dump: reading transforms
pg_dump: reading table inheritance information
pg_dump: reading event triggers
pg_dump: finding extension tables
pg_dump: finding inheritance relationships
pg_dump: reading column info for interesting tables
pg_dump: finding the columns and types of table "test.large_test"
pg_dump: flagging inherited columns in subtables
pg_dump: reading indexes
pg_dump: reading extended statistics
pg_dump: reading constraints
pg_dump: reading triggers
pg_dump: reading rewrite rules
pg_dump: reading policies
pg_dump: reading row security enabled for table "test.large_test"
pg_dump: reading policies for table "test.large_test"
pg_dump: reading publications
pg_dump: reading publication membership
pg_dump: reading publication membership for table "test.large_test"
pg_dump: reading subscriptions
pg_dump: reading dependency data
pg_dump: saving encoding = UTF8
pg_dump: saving standard_conforming_strings = on
pg_dump: saving search_path = 
pg_dump: dumping contents of table "test.large_test"
pg_dump: could not stat file
"O:\postgres-server3\test#test#large_test/2793.dat.gz": Unknown error

=?utf-8?q?PG_Bug_reporting_form?= <noreply@postgresql.org> writes:
> When I pg_dump a large table (> 35 GB), I get the following error message:

> pg_dump: could not stat file
> "O:\postgres-server3\test#test#large_test/2793.dat.gz": Unknown error

> The dump however appears to restore correctly.

Hm, I assume this shows up just at the end of the dump run?

The only plausible match for that error string is in the
fsync_dir_recurse() processing that tries to force all the output files
down to disk before reporting that the dump is complete.  As long as
you don't have an OS crash right afterwards, failure to fsync is
harmless.  Still, it's weird.

Does it vary with the size of the dumped table?  What if you remove
the parallelism (no --jobs option)?

            regards, tom lane

Mark Lai <mark.lai@integrafec.com> writes:
> I ran the dump on the large table with no jobs flag and got the same error.
> ...
> The dump was successful on a small table.

Weird indeed.  Can any Windows developers reproduce this and poke into it?

I have a sneaking suspicion that this is related to Windows' known issues
with concurrently-opened files, but it's pretty hard to see why there
would be a dependency on the size of the file.

            regards, tom lane

On Fri, Aug 17, 2018 at 10:53:11AM -0400, Tom Lane wrote:
> Mark Lai <mark.lai@integrafec.com> writes:
>> I ran the dump on the large table with no jobs flag and got the same error.
>> ...
>> The dump was successful on a small table.
>
> Weird indeed.  Can any Windows developers reproduce this and poke into it?
>
> I have a sneaking suspicion that this is related to Windows' known issues
> with concurrently-opened files, but it's pretty hard to see why there
> would be a dependency on the size of the file.

When it comes to pg_dump, the error message reported seems to come from
src/common/file_utils.c, in walkdir when processing links.  On Windows
we map lstat() to stat(), which is itself pgwin32_safestat().

If you use pg_dump --no-sync, the error could be bypassed but that's
hardly a fix.  That could be a failure on GetFileAttributeEx().  Which
file system are you using?
--
Michael