Thread: PostgreSQL 12 - ERROR: could not rename file "pg_logical/snapshots/1A-7C00D890.snap.13594.tmp" to "pg_logical/snapshots/1A-7C00D890.snap": No space left on device

On 6/30/20 11:03 AM, FOUTE K. Jaurès wrote:
> Hi everyone,
> 
> I am use a native logical replication in PostgreSQL 12 on Ubuntu 16.04 
> and all is working fine until i faced this error below.

 From below it looks like you are using pg_logical which is a third 
party package. It is what the builtin logical replication is derived 
from, but it is not the same thing. So what version of pg_logical are 
you using?

> 
> ERROR:  could not rename file 
> "pg_logical/snapshots/1A-7C00D890.snap.13594.tmp" to 
> "pg_logical/snapshots/1A-7C00D890.snap": No space left on device
> 
> Is there someone who faced this problem?
> Any idea how I can solve it ?
> 
> BTW: I don't have any problem with space.

The error says you do.
Where is pg_logical/snapshots/ mounted?
Are there specific restrictions on that mount?

> 
> Best Regard
> 
> -- 
> Jaurès FOUTE


-- 
Adrian Klaver
adrian.klaver@aklaver.com

On 6/30/20 1:47 PM, FOUTE K. Jaurès wrote:
> 
> 
> Le mar. 30 juin 2020 à 21:23, Bruce Momjian <bruce@momjian.us 
> <mailto:bruce@momjian.us>> a écrit :
> 
>     On Tue, Jun 30, 2020 at 01:16:58PM -0700, Adrian Klaver wrote:
>      > On 6/30/20 11:03 AM, FOUTE K. Jaurès wrote:
>      > > Hi everyone,
>      > >
>      > > I am use a native logical replication in PostgreSQL 12 on
>     Ubuntu 16.04
>      > > and all is working fine until i faced this error below.
>      >
>      > From below it looks like you are using pg_logical which is a
>     third party
>      > package. It is what the builtin logical replication is derived
>     from, but it
>      > is not the same thing. So what version of pg_logical are you using?
> 
> I only install Postgresql 12. I dont use pg_logical

My mistake, I remembered the location for the logical files wrong.


> 
> /var/lib/postgresql/12/main/pg_logical (default location)

 From /var/lib/postgresql/12/main/pg_logical what does du -sh  show?
Please show results as text, you can copy from terminal screen.


> 
>      > Are there specific restrictions on that mount?
> 
> image.png
> 
> 
>     I would also look at your kernel log.
> 
>     -- 
>        Bruce Momjian  <bruce@momjian.us <mailto:bruce@momjian.us>>
>     https://momjian.us
>        EnterpriseDB https://enterprisedb.com
> 
>        The usefulness of a cup is in its emptiness, Bruce Lee
> 
> 
> 
> -- 
> Jaurès FOUTE


-- 
Adrian Klaver
adrian.klaver@aklaver.com

FOUTE K. Jaurès wrote:

> Hi everyone,
> 
> I am use a native logical replication in PostgreSQL 12 on Ubuntu 16.04 and
> all is working fine until i faced this error below.
> 
> ERROR:  could not rename file
> "pg_logical/snapshots/1A-7C00D890.snap.13594.tmp" to
> "pg_logical/snapshots/1A-7C00D890.snap": No space left on device
> 
> Is there someone who faced this problem?
> Any idea how I can solve it ?
> 
> BTW: I don't have any problem with space.
> 
> Best Regard
> 
> -- 
> Jaurès FOUTE

If you really haven't run out of space,
you might have run out of inodes.
The -i option of df should show you.
100,000 empty files could cause that.
I wouldn't think that renaming would
require a new inode, but I also wouldn't
think that renaming would require any
more space on a file system.

cheers,
raf

On 6/30/20 10:51 PM, FOUTE K. Jaurès wrote:
> 
> 

> 
>       From /var/lib/postgresql/12/main/pg_logical what does du -sh  show?
>     Please show results as text, you can copy from terminal screen.
> 
> root@hybride:/var/lib/postgresql/12/main/pg_logical# du -sh
> 16K .
> 

Has the problem occurred again?


-- 
Adrian Klaver
adrian.klaver@aklaver.com

On 2020-Jun-30, FOUTE K. Jaurès wrote:

> Hi everyone,
> 
> I am use a native logical replication in PostgreSQL 12 on Ubuntu 16.04 and
> all is working fine until i faced this error below.
> 
> ERROR:  could not rename file
> "pg_logical/snapshots/1A-7C00D890.snap.13594.tmp" to
> "pg_logical/snapshots/1A-7C00D890.snap": No space left on device

What are your fsync and wal_sync_method settings?

-- 
Álvaro Herrera                https://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

On 7/1/20 11:21 AM, FOUTE K. Jaurès wrote:
> 

> 
>     Has the problem occurred again?
> 
> Yes
> 2020-07-01 19: 19: 21.868 WAT [24204] ERROR: could not receive data from 
> WAL stream: ERROR: could not rename file "pg_logical / snapshots / 
> 1E-8567EB0.snap.22195.tmp" to "pg_logical / snapshots / 
> 1E-8567EB0.snap": No space left on device

What are the permissions on?:

pg_logical/snapshots/

> 
> 
> 
>     -- 
>     Adrian Klaver
>     adrian.klaver@aklaver.com <mailto:adrian.klaver@aklaver.com>
> 
> 
> 
> -- 
> Jaurès FOUTE
> Technology Consultant
> ISNOV SARL - Business Technology Consulting
> Tel: +237 79395671 / +237 96248793
> Email: _jauresfoute@gmail.com <mailto:jauresfoute@gmail.com>_,
> jauresmelkiore@yahoo.fr <mailto:jauresmelkiore@yahoo.fr>
> Compte Skype: jauresmelkiore


-- 
Adrian Klaver
adrian.klaver@aklaver.com

On 7/1/20 11:21 AM, FOUTE K. Jaurès wrote:
> 
> 
> Le mer. 1 juil. 2020 à 15:45, Adrian Klaver <adrian.klaver@aklaver.com 
> <mailto:adrian.klaver@aklaver.com>> a écrit :
> 
>     On 6/30/20 10:51 PM, FOUTE K. Jaurès wrote:
>      >
>      >
> 
>      >
>      >       From /var/lib/postgresql/12/main/pg_logical what does du
>     -sh  show?
>      >     Please show results as text, you can copy from terminal screen.
>      >
>      > root@hybride:/var/lib/postgresql/12/main/pg_logical# du -sh
>      > 16K .
>      >
> 
>     Has the problem occurred again?
> 
> Yes
> 2020-07-01 19: 19: 21.868 WAT [24204] ERROR: could not receive data from 
> WAL stream: ERROR: could not rename file "pg_logical / snapshots / 
> 1E-8567EB0.snap.22195.tmp" to "pg_logical / snapshots / 
> 1E-8567EB0.snap": No space left on device

Also look in system log for any events around 2020-07-01 19: 19

> 
> 
> 
>     -- 
>     Adrian Klaver
>     adrian.klaver@aklaver.com <mailto:adrian.klaver@aklaver.com>
> 
> 
> 
> -- 
> Jaurès FOUTE
> Technology Consultant
> ISNOV SARL - Business Technology Consulting
> Tel: +237 79395671 / +237 96248793
> Email: _jauresfoute@gmail.com <mailto:jauresfoute@gmail.com>_,
> jauresmelkiore@yahoo.fr <mailto:jauresmelkiore@yahoo.fr>
> Compte Skype: jauresmelkiore


-- 
Adrian Klaver
adrian.klaver@aklaver.com

On 7/1/20 2:21 PM, FOUTE K. Jaurès wrote:
> 
> 
> Le mer. 1 juil. 2020 à 22:15, Adrian Klaver <adrian.klaver@aklaver.com 
> <mailto:adrian.klaver@aklaver.com>> a écrit :
> 
>     On 7/1/20 11:21 AM, FOUTE K. Jaurès wrote:
>      >
> 
>      >
>      >     Has the problem occurred again?
>      >
>      > Yes
>      > 2020-07-01 19: 19: 21.868 WAT [24204] ERROR: could not receive
>     data from
>      > WAL stream: ERROR: could not rename file "pg_logical / snapshots /
>      > 1E-8567EB0.snap.22195.tmp" to "pg_logical / snapshots /
>      > 1E-8567EB0.snap": No space left on device
> 
>     What are the permissions on?:
> 
>     pg_logical/snapshots/
> 
> postgres
> root@hybride:/var/lib/postgresql/12/main/pg_logical# ll
> total 20
> drwx------  4 postgres postgres 4096 juil.  1 20:21 ./
> drwx------ 19 postgres postgres 4096 juil.  1 20:16 ../
> drwx------  2 postgres postgres 4096 mars  31 11:58 mappings/
> -rw-------  1 postgres postgres   56 juil.  1 20:21 replorigin_checkpoint
> drwx------  2 postgres postgres 4096 mars  31 11:58 snapshots/


Well replication is between two instances, so are you sure you are 
looking at the correct data directory?



> -- 
> Jaurès FOUTE
> Technology Consultant
> ISNOV SARL - Business Technology Consulting
> Tel: +237 79395671 / +237 96248793
> Email: _jauresfoute@gmail.com <mailto:jauresfoute@gmail.com>_,
> jauresmelkiore@yahoo.fr <mailto:jauresmelkiore@yahoo.fr>
> Compte Skype: jauresmelkiore


-- 
Adrian Klaver
adrian.klaver@aklaver.com

FOUTE K. Jaurès wrote:

> Le mer. 1 juil. 2020 à 00:11, raf <raf@raf.org> a écrit :
> 
> > FOUTE K. Jaurčs wrote:
> >
> > > Hi everyone,
> > >
> > > I am use a native logical replication in PostgreSQL 12 on Ubuntu 16.04
> > and
> > > all is working fine until i faced this error below.
> > >
> > > ERROR:  could not rename file
> > > "pg_logical/snapshots/1A-7C00D890.snap.13594.tmp" to
> > > "pg_logical/snapshots/1A-7C00D890.snap": No space left on device
> > >
> > > Is there someone who faced this problem?
> > > Any idea how I can solve it ?
> > >
> > > BTW: I don't have any problem with space.
> > >
> > > Best Regard
> > >
> > > --
> > > Jaurčs FOUTE
> >
> > If you really haven't run out of space,
> > you might have run out of inodes.
> > The -i option of df should show you.
> > 100,000 empty files could cause that.
> > I wouldn't think that renaming would
> > require a new inode, but I also wouldn't
> > think that renaming would require any
> > more space on a file system.
> 
> 
> The result of df -i
> 
> Sys. de fichiers   Inœuds IUtil.   ILibre IUti% Monté sur
> udev              3065149    433  3064716    1% /dev
> tmpfs             3072780    665  3072115    1% /run
> /dev/sdb2        59973632 356029 59617603    1% /
> tmpfs             3072780      7  3072773    1% /dev/shm
> tmpfs             3072780     10  3072770    1% /run/lock
> tmpfs             3072780     17  3072763    1% /sys/fs/cgroup
> /dev/sdb1               0      0        0     - /boot/efi
> tmpfs             3072780     19  3072761    1% /run/user/108
> tmpfs             3072780      5  3072775    1% /run/user/1001
> tmpfs             3072780      5  3072775    1% /run/user/1000

So that's not it. It would take ~60 million
files to fill up your / inode table. I can't
think of another explanation for that error
message if df without -i also shows free space.

I tried googling for "No space left on device"
and everything suggested checking inodes with "df -i".
One suggested looking for files that were deleted
but that were still open by a process (can be found
with "lsof / | grep deleted") but if that were the
problem, df would show a full disk (but du wouldn't
be able to account for it).

Could it be some other "device"? like shared memory
segment space or IPC resources? That seems unlikely if
the operation was just a file system rename. Perhaps
strace/stap could help check if it really was a file
system rename that went wrong (if there's any reason to
doubt it). It does seem to clearly be a file rename
though.

If there any chance that the drive is failing? But
you'd expect "I/O error" messages if that were the case.

Is it on a journalling file system and could the journal
be the device that is full? I don't know how to check
that (or if it even makes sense).

cheers,
raf

raf <raf@raf.org> writes:
> FOUTE K. Jaurès wrote:
>> The result of df -i
>> 
>> Sys. de fichiers   Inœuds IUtil.   ILibre IUti% Monté sur
>> udev              3065149    433  3064716    1% /dev
>> tmpfs             3072780    665  3072115    1% /run
>> /dev/sdb2        59973632 356029 59617603    1% /
>> tmpfs             3072780      7  3072773    1% /dev/shm
>> tmpfs             3072780     10  3072770    1% /run/lock
>> tmpfs             3072780     17  3072763    1% /sys/fs/cgroup
>> /dev/sdb1               0      0        0     - /boot/efi
>> tmpfs             3072780     19  3072761    1% /run/user/108
>> tmpfs             3072780      5  3072775    1% /run/user/1001
>> tmpfs             3072780      5  3072775    1% /run/user/1000

> So that's not it. It would take ~60 million
> files to fill up your / inode table. I can't
> think of another explanation for that error
> message if df without -i also shows free space.

I'm going to take a shot in the dark and ask if the root file system
is XFS.  It doesn't take too much googling to find out that XFS has a
reputation for reporting ENOSPC when there seems to be more than enough
room left.  Apparently, it can do so as a result of fragmentation problems
even though the disk as a whole has lots of space --- for one cautionary
example see this thread:

https://www.spinics.net/lists/linux-xfs/msg22856.html

typo-ishly titled "ENSOPC on a 10% used disk".  It looks like the XFS crew
installed a fix for the underlying bug ... but that thread is from 2018
and you're running a 2015 Ubuntu release.

Or, to cut to the chase: maybe updating to a less hoary kernel would help.
If you are stuck with this Ubuntu release for some reason, consider using
a less bleeding-edge-at-the-time file system.

            regards, tom lane

=?UTF-8?Q?FOUTE_K=2E_Jaur=C3=A8s?= <jauresfoute@gmail.com> writes:
> After many tests of proposal solutions, I am not able to solve the issue.
> Seems that It is a bug ?

I agree.  But it's not *our* bug: Postgres is merely reporting what the
OS told it.  File a kernel bug with your OS vendor.

            regards, tom lane