Re: PostgreSQL 12 - ERROR: could not rename file "pg_logical/snapshots/1A-7C00D890.snap.13594.tmp" to "pg_logical/snapshots/1A-7C00D890.snap": No space left on device - Mailing list pgsql-general

From raf
Subject Re: PostgreSQL 12 - ERROR: could not rename file "pg_logical/snapshots/1A-7C00D890.snap.13594.tmp" to "pg_logical/snapshots/1A-7C00D890.snap": No space left on device
Date
Msg-id 20200702000622.ehwnezoyu2valp5t@raf.org
Whole thread Raw
In response to Re: PostgreSQL 12 - ERROR: could not rename file "pg_logical/snapshots/1A-7C00D890.snap.13594.tmp" to "pg_logical/snapshots/1A-7C00D890.snap": No space left on device  (FOUTE K. Jaurès <jauresfoute@gmail.com>)
Responses Re: PostgreSQL 12 - ERROR: could not rename file "pg_logical/snapshots/1A-7C00D890.snap.13594.tmp" to "pg_logical/snapshots/1A-7C00D890.snap": No space left on device
List pgsql-general
FOUTE K. Jaurès wrote:

> Le mer. 1 juil. 2020 à 00:11, raf <raf@raf.org> a écrit :
> 
> > FOUTE K. Jaurčs wrote:
> >
> > > Hi everyone,
> > >
> > > I am use a native logical replication in PostgreSQL 12 on Ubuntu 16.04
> > and
> > > all is working fine until i faced this error below.
> > >
> > > ERROR:  could not rename file
> > > "pg_logical/snapshots/1A-7C00D890.snap.13594.tmp" to
> > > "pg_logical/snapshots/1A-7C00D890.snap": No space left on device
> > >
> > > Is there someone who faced this problem?
> > > Any idea how I can solve it ?
> > >
> > > BTW: I don't have any problem with space.
> > >
> > > Best Regard
> > >
> > > --
> > > Jaurčs FOUTE
> >
> > If you really haven't run out of space,
> > you might have run out of inodes.
> > The -i option of df should show you.
> > 100,000 empty files could cause that.
> > I wouldn't think that renaming would
> > require a new inode, but I also wouldn't
> > think that renaming would require any
> > more space on a file system.
> 
> 
> The result of df -i
> 
> Sys. de fichiers   Inœuds IUtil.   ILibre IUti% Monté sur
> udev              3065149    433  3064716    1% /dev
> tmpfs             3072780    665  3072115    1% /run
> /dev/sdb2        59973632 356029 59617603    1% /
> tmpfs             3072780      7  3072773    1% /dev/shm
> tmpfs             3072780     10  3072770    1% /run/lock
> tmpfs             3072780     17  3072763    1% /sys/fs/cgroup
> /dev/sdb1               0      0        0     - /boot/efi
> tmpfs             3072780     19  3072761    1% /run/user/108
> tmpfs             3072780      5  3072775    1% /run/user/1001
> tmpfs             3072780      5  3072775    1% /run/user/1000

So that's not it. It would take ~60 million
files to fill up your / inode table. I can't
think of another explanation for that error
message if df without -i also shows free space.

I tried googling for "No space left on device"
and everything suggested checking inodes with "df -i".
One suggested looking for files that were deleted
but that were still open by a process (can be found
with "lsof / | grep deleted") but if that were the
problem, df would show a full disk (but du wouldn't
be able to account for it).

Could it be some other "device"? like shared memory
segment space or IPC resources? That seems unlikely if
the operation was just a file system rename. Perhaps
strace/stap could help check if it really was a file
system rename that went wrong (if there's any reason to
doubt it). It does seem to clearly be a file rename
though.

If there any chance that the drive is failing? But
you'd expect "I/O error" messages if that were the case.

Is it on a journalling file system and could the journal
be the device that is full? I don't know how to check
that (or if it even makes sense).

cheers,
raf




pgsql-general by date:

Previous
From: Tom Lane
Date:
Subject: Re: Different results from identical matviews
Next
From: Tom Lane
Date:
Subject: Re: PostgreSQL 12 - ERROR: could not rename file "pg_logical/snapshots/1A-7C00D890.snap.13594.tmp" to "pg_logical/snapshots/1A-7C00D890.snap": No space left on device