Thread: How to create directory format backup
Hi!
Creating backup in directory format using
pg_dump -f "sba" -Fdirectory --jobs=32 --verbose sba
throws error
pg_dump: error: could not stat file "sba/282168.data.gz": value too large
How to fix it ?
Server is Postgres 12 running in Debian Linux 10 under WSL
Client is pg_dump from Postgres 15 running in Windows 11
Andrus
On Wed, Feb 08, 2023 at 05:00:10PM +0200, Andrus wrote: > Hi! > > Creating backup in directory format using > > pg_dump -f "sba" -Fdirectory --jobs=32 --verbose sba > > throws error > > pg_dump: error: could not stat file "sba/282168.data.gz": value too > large > > How to fix it ? > > Server is Postgres 12 running in Debian Linux 10 under WSL > Client is pg_dump from Postgres 15 running in Windows 11 Looks like your filesystem on client is having limits on file sizes. Use better filesystem, or just dump on linux, it's filesystems usually don't hit these limits. Best regards, depesz
>Looks like your filesystem on client is having limits on file sizes. Use better filesystem, or just dump on linux, it's filesystems usually don't hit these limits.
This file size is only 6.2 GB. If custom format is used, pg_dump creates large file without problems. There are no file size limits. Error message is not about this.
Backup client is running in Windows 11 and this cannot changed.
How to create backup in format from which tables can selectively restored?
Posted also in https://stackoverflow.com/questions/75387616/how-to-make-directory-format-backup
Andrus.
> On 08/02/2023 21:59 CET Andrus <kobruleht2@hot.ee> wrote: > > How to create backup in format from which tables can selectively restored? Dump as custom-format archive (-F custom) and use that with pg_restore and options --table or --list/--use-list to select what should be restored. -- Erik
Hi!
How to create backup in format from which tables can selectively restored?Dump as custom-format archive (-F custom) and use that with pg_restore and options --table or --list/--use-list to select what should be restored.
How to select tables interactively like pgAdmin allows to select when directory format is used ?
Database contains hundreds of schemas. I need to restore public and other other schema.
Whole backup file is scanned to restore only two schemas. It takes lot of time.
Also directory format allows to use all cores with --jobs=32 parameter. Dump and partial restore using custom format are much slower.
Andrus.
-- Erik
> On 08/02/2023 22:37 CET Andrus <kobruleht2@hot.ee> wrote: > > > > How to create backup in format from which tables can selectively > > > restored? > > > > > Dump as custom-format archive (-F custom) and use that with pg_restore > > and options --table or --list/--use-list to select what should be > > restored. > > > How to select tables interactively like pgAdmin allows to select when > directory format is used ? > > Database contains hundreds of schemas. I need to restore public and other > other schema. > > Whole backup file is scanned to restore only two schemas. It takes lot of > time. pg_dump also accepts options --table and --schema to only dump what you need. > Also directory format allows to use all cores with --jobs=32 parameter. > Dump and partial restore using custom format are much slower. Run multiple pg_dump processes in parallel where each processes a subset of tables with the options mentioned above. -- Erik
Andrus <kobruleht2@hot.ee> writes:
> This file size is only 6.2 GB. If custom format is used, pg_dump creates
> large file without problems. There are no file size limits. Error
> message is not about this.
Are you certain that the pg_dump you're using is v15, and not something
pre-v14? We got rid of the 4GB limit on stat() results in v14,
or so we thought.
Anyway, as far as I can find pg_dump can only reach that error message
via fsync_dir_recurse(), which suggests that a workaround might be
to add --no-sync so it doesn't try to fsync its output.
regards, tom lane