Home > mailing lists

Thread: pg_basebackup Compression Problem

pg_basebackup Compression Problem

From

Ankur Kaushik

Date:

20 December 2017, 13:33:48

Facing problem in taking pg_basebackup .

File Reaches till 2 GB then failes and give below error

pg_basebackup: could not write to compressed file "/backup//base.tar.gz": File too large

Command :

pg_basebackup -h127.0.0.1 -U base_backup_user -D /backup/inc_backup -Ft -z -P

Re: pg_basebackup Compression Problem

From

Keith

Date:

20 December 2017, 13:47:40

On Wed, Dec 20, 2017 at 2:33 AM, Ankur Kaushik <ankurkaushik@gmail.com> wrote:

Facing problem in taking pg_basebackup .

File Reaches till 2 GB then failes and give below error

pg_basebackup: could not write to compressed file "/backup//base.tar.gz": File too large

Command :

pg_basebackup -h127.0.0.1 -U base_backup_user -D /backup/inc_backup -Ft -z -P

Are you writing the backup to a FAT32 file system? Those have a max individual file size limit of about 2GB.

Keith

Re: pg_basebackup Compression Problem

From

Shreeyansh Dba

Date:

20 December 2017, 13:52:20

On Wed, Dec 20, 2017 at 1:03 PM, Ankur Kaushik <ankurkaushik@gmail.com> wrote:

Facing problem in taking pg_basebackup .

File Reaches till 2 GB then failes and give below error

pg_basebackup: could not write to compressed file "/backup//base.tar.gz": File too large

Command :

pg_basebackup -h127.0.0.1 -U base_backup_user -D /backup/inc_backup -Ft -z -P

It is not the issue with pg_basebackup you are using

It seems, this issue might be with storage location where you are trying to take backup(i.e. /backup)

Re: pg_basebackup Compression Problem

From

Ankur Kaushik

Date:

20 December 2017, 13:52:33

OS Config :

Linux 2.6.32-573.12.1.el6.i686 #1 SMP Tue Dec 15 18:25:17 UTC 2015 i686 i686 i386 GNU/Linux

CentOS release 6.7 (Final)

Kernel \r on an \m

When We do with manual gzip to to generated tar it works but problem when we do with pg_basebackup

On Wed, Dec 20, 2017 at 1:17 PM, Keith <keith@keithf4.com> wrote:

On Wed, Dec 20, 2017 at 2:33 AM, Ankur Kaushik <ankurkaushik@gmail.com> wrote:

Facing problem in taking pg_basebackup .

File Reaches till 2 GB then failes and give below error

pg_basebackup: could not write to compressed file "/backup//base.tar.gz": File too large

Command :

pg_basebackup -h127.0.0.1 -U base_backup_user -D /backup/inc_backup -Ft -z -P

Are you writing the backup to a FAT32 file system? Those have a max individual file size limit of about 2GB.

Keith

Re: pg_basebackup Compression Problem

From

Ankur Kaushik

Date:

20 December 2017, 13:54:03

Storage Location is fine :

/dev/sda3 451G 258G 170G 61% /

tmpfs 3.9G 228K 3.9G 1% /dev/shm

/dev/sda1 190M 104M 76M 58% /boot

On Wed, Dec 20, 2017 at 1:22 PM, Ankur Kaushik <ankurkaushik@gmail.com> wrote:

OS Config :

Linux 2.6.32-573.12.1.el6.i686 #1 SMP Tue Dec 15 18:25:17 UTC 2015 i686 i686 i386 GNU/Linux

CentOS release 6.7 (Final)
Kernel \r on an \m

When We do with manual gzip to to generated tar it works but problem when we do with pg_basebackup

On Wed, Dec 20, 2017 at 1:17 PM, Keith <keith@keithf4.com> wrote:

On Wed, Dec 20, 2017 at 2:33 AM, Ankur Kaushik <ankurkaushik@gmail.com> wrote:

Facing problem in taking pg_basebackup .

File Reaches till 2 GB then failes and give below error

pg_basebackup: could not write to compressed file "/backup//base.tar.gz": File too large

Command :

pg_basebackup -h127.0.0.1 -U base_backup_user -D /backup/inc_backup -Ft -z -P

Are you writing the backup to a FAT32 file system? Those have a max individual file size limit of about 2GB.

Keith

Re: pg_basebackup Compression Problem

From

Ankur Kaushik

Date:

20 December 2017, 13:54:36

Postgres Version : psql (PostgreSQL) 9.4.8

On Wed, Dec 20, 2017 at 1:24 PM, Ankur Kaushik <ankurkaushik@gmail.com> wrote:

Storage Location is fine :

/dev/sda3 451G 258G 170G 61% /
tmpfs 3.9G 228K 3.9G 1% /dev/shm
/dev/sda1 190M 104M 76M 58% /boot

On Wed, Dec 20, 2017 at 1:22 PM, Ankur Kaushik <ankurkaushik@gmail.com> wrote:

OS Config :

Linux 2.6.32-573.12.1.el6.i686 #1 SMP Tue Dec 15 18:25:17 UTC 2015 i686 i686 i386 GNU/Linux

CentOS release 6.7 (Final)
Kernel \r on an \m

When We do with manual gzip to to generated tar it works but problem when we do with pg_basebackup

On Wed, Dec 20, 2017 at 1:17 PM, Keith <keith@keithf4.com> wrote:

On Wed, Dec 20, 2017 at 2:33 AM, Ankur Kaushik <ankurkaushik@gmail.com> wrote:

Facing problem in taking pg_basebackup .

File Reaches till 2 GB then failes and give below error

pg_basebackup: could not write to compressed file "/backup//base.tar.gz": File too large

Command :

pg_basebackup -h127.0.0.1 -U base_backup_user -D /backup/inc_backup -Ft -z -P

Are you writing the backup to a FAT32 file system? Those have a max individual file size limit of about 2GB.

Keith

Re: pg_basebackup Compression Problem

From

Shreeyansh Dba

Date:

20 December 2017, 14:01:58

On Wed, Dec 20, 2017 at 1:22 PM, Shreeyansh Dba <shreeyansh2014@gmail.com> wrote:

On Wed, Dec 20, 2017 at 1:03 PM, Ankur Kaushik <ankurkaushik@gmail.com> wrote:

Facing problem in taking pg_basebackup .

File Reaches till 2 GB then failes and give below error

pg_basebackup: could not write to compressed file "/backup//base.tar.gz": File too large

Command :

pg_basebackup -h127.0.0.1 -U base_backup_user -D /backup/inc_backup -Ft -z -P

It is not the issue with pg_basebackup you are using
It seems, this issue might be with storage location where you are trying to take backup(i.e. /backup)

In addition to this,

Your storage might be having limitations for not accepting file storage of more than 2GB, hence this error.

--

Re: pg_basebackup Compression Problem

From

Magnus Hagander

Date:

20 December 2017, 14:36:16

On Wed, Dec 20, 2017 at 8:52 AM, Ankur Kaushik <ankurkaushik@gmail.com> wrote:

OS Config :

Linux 2.6.32-573.12.1.el6.i686 #1 SMP Tue Dec 15 18:25:17 UTC 2015 i686 i686 i386 GNU/Linux

CentOS release 6.7 (Final)
Kernel \r on an \m

When We do with manual gzip to to generated tar it works but problem when we do with pg_basebackup

This is a 32-bit operating system. pg_basebackup does not use whatever features are in libz for dealing with files larger than 4Gb on a 32-bit operating system, so that's your issue right there.

How did you find yourself with a 32-bit operating system in 2017? I think the general idea when we built that into pg_basebackup was that nobody with anything that big would be running a 32-bit operating system those days, and certainly not now..

Your workaround for that is to make pg_basebackup write to stdout, and use an external gzip in a pipe.

//Magnus

Re: pg_basebackup Compression Problem

From

Ankur Kaushik

Date:

20 December 2017, 15:37:06

Thanks , This could be the Case external gzip is working fine . Also I will check same on 64 bit OS

On Wed, Dec 20, 2017 at 2:06 PM, Magnus Hagander <magnus@hagander.net> wrote:

On Wed, Dec 20, 2017 at 8:52 AM, Ankur Kaushik <ankurkaushik@gmail.com> wrote:

OS Config :

Linux 2.6.32-573.12.1.el6.i686 #1 SMP Tue Dec 15 18:25:17 UTC 2015 i686 i686 i386 GNU/Linux

CentOS release 6.7 (Final)
Kernel \r on an \m

When We do with manual gzip to to generated tar it works but problem when we do with pg_basebackup

This is a 32-bit operating system. pg_basebackup does not use whatever features are in libz for dealing with files larger than 4Gb on a 32-bit operating system, so that's your issue right there.

How did you find yourself with a 32-bit operating system in 2017? I think the general idea when we built that into pg_basebackup was that nobody with anything that big would be running a 32-bit operating system those days, and certainly not now..

Your workaround for that is to make pg_basebackup write to stdout, and use an external gzip in a pipe.

//Magnus