Thread: PANIC: could not create file "pg_xlog/xlogtemp.7884": No space left on device

On 25/09/2014 11:44, Roopeshakumar Narayansa Shalgar (rshalgar) wrote:
> Hi,
>
>
>
> I am using version 9.3.1 and see the “no space device error” even though
> there is enough space (99% free) on my disk.
>
>
>
> Error: PANIC:  could not create file "pg_xlog/xlogtemp.7884": No space
> left on device
>
>
>
> I want to know whether this is a bug and if yes whether fix or
> workaround is available.

It's not a bug - you're out of disk space. As it happens, I saw a blog
post yesterday on precisely this problem, and approaches to fixing it:

http://blog.endpoint.com/2014/09/pgxlog-disk-space-problem-on-postgres.html


HTH,

Ray.


--
Raymond O'Donnell :: Galway :: Ireland
rod@iol.ie

On 25/09/2014 11:44, Roopeshakumar Narayansa Shalgar (rshalgar) wrote:
> Hi,
>
>
>
> I am using version 9.3.1 and see the “no space device error” even though
> there is enough space (99% free) on my disk.
>
>
>
> Error: PANIC:  could not create file "pg_xlog/xlogtemp.7884": No space
> left on device
>
>
>
> I want to know whether this is a bug and if yes whether fix or
> workaround is available.

It's not a bug - you're out of disk space. As it happens, I saw a blog
post yesterday on precisely this problem, and approaches to fixing it:

http://blog.endpoint.com/2014/09/pgxlog-disk-space-problem-on-postgres.html


HTH,

Ray.


--
Raymond O'Donnell :: Galway :: Ireland
rod@iol.ie

Roopeshakumar Narayansa Shalgar (rshalgar) wrote
> bash-3.2# du -h
> 4.0K    ./pg_stat
> 6.2M    ./base/12026
> 198M    ./base/16384
> 6.3M    ./base/12031
> 6.3M    ./base/1
> 216M    ./base
> 24K     ./pg_stat_tmp
> [...]
> 129M    ./pg_xlog
> 4.0K    ./pg_tblspc
> 12K     ./pg_notify
> 4.0K    ./pg_twophase
> 128K    ./pg_clog
> 4.0K    ./pg_subtrans

Did you maybe mean to provide the output of "df" instead of "du"?  All you
show above is the size of the postgres data directory - which I doubt is the
only thing on the partition in question.

David J.





--
View this message in context:
http://postgresql.1045698.n5.nabble.com/PANIC-could-not-create-file-pg-xlog-xlogtemp-7884-No-space-left-on-device-tp5820686p5820695.html
Sent from the PostgreSQL - general mailing list archive at Nabble.com.

Thanks Raymond.

But I think the symptoms that I am seeing are different though I get the
no disk space issue in the following ways:

1. There is enough disk space on my device (output of df -h), 300M is used from 56G space.

/dev/mapper/data1-object--cache
                       58G  300M   58G   1% /object-cache1



2.  ls -rtl pg_xlog:

drwx------ 2 postgres 75     4096 Sep 28 03:45 archive_status
-rw------- 1 postgres 75 16777216 Sep 28 04:33 00000001000000000000000B
-rw------- 1 postgres 75 16777216 Sep 28 04:34 00000001000000000000000C

( I see the WAL xlog files are getting created and the older ones are getting deleted correctly and
   they are using hardly 32MB from a partition of 58G so where is the question of no space)

3. Could this be related to:
         http://www.postgresql.org/message-id/20140414014442.15385.74268@wrigleys.postgresql.org.

Here there is mention of 'long writes and short writes', will trying out latest version of postgress help?

4. I tried with 'archive_command = /usr/bin.true' but still I face the ""pg_xlog/xlogtemp.7884": No space left on
device"issue. 

5. Once I  hit ""pg_xlog/xlogtemp.7884": No space left on device" I start seeing it for other files as well:

    "WARNING:  could not create relation-cache initialization file "base/16384/pg_internal.init.13715": No space left
     on device DETAIL:  Continuing anyway, but there's something wrong."

6. I am seeing this issue when I create around 60,000 to 70,000 files (each of size 4k) in the data base.


Appreciate any quick help on this.

--
Thanks and regards,
Rupesh


-----Original Message-----
From: Raymond O'Donnell [mailto:rod@iol.ie]
Sent: Friday, September 26, 2014 9:41 PM
To: Roopeshakumar Narayansa Shalgar (rshalgar); 'PostgreSQL'
Subject: Re: [GENERAL] PANIC: could not create file "pg_xlog/xlogtemp.7884": No space left on device

On 25/09/2014 11:44, Roopeshakumar Narayansa Shalgar (rshalgar) wrote:
> Hi,
>
>
>
> I am using version 9.3.1 and see the "no space device error" even though
> there is enough space (99% free) on my disk.
>
>
>
> Error: PANIC:  could not create file "pg_xlog/xlogtemp.7884": No space
> left on device
>
>
>
> I want to know whether this is a bug and if yes whether fix or
> workaround is available.

It's not a bug - you're out of disk space. As it happens, I saw a blog
post yesterday on precisely this problem, and approaches to fixing it:

http://blog.endpoint.com/2014/09/pgxlog-disk-space-problem-on-postgres.html


HTH,

Ray.


--
Raymond O'Donnell :: Galway :: Ireland
rod@iol.ie

On 09/28/2014 05:52 AM, Roopeshakumar Narayansa Shalgar (rshalgar) wrote:
> Thanks Raymond.
>
> But I think the symptoms that I am seeing are different though I get the
> no disk space issue in the following ways:
>
> 1. There is enough disk space on my device (output of df -h), 300M is used from 56G space.
>
> /dev/mapper/data1-object--cache
>                         58G  300M   58G   1% /object-cache1
>
>
>
> 2.  ls -rtl pg_xlog:
>
> drwx------ 2 postgres 75     4096 Sep 28 03:45 archive_status
> -rw------- 1 postgres 75 16777216 Sep 28 04:33 00000001000000000000000B
> -rw------- 1 postgres 75 16777216 Sep 28 04:34 00000001000000000000000C
>
> ( I see the WAL xlog files are getting created and the older ones are getting deleted correctly and
>     they are using hardly 32MB from a partition of 58G so where is the question of no space)
>
> 3. Could this be related to:
>           http://www.postgresql.org/message-id/20140414014442.15385.74268@wrigleys.postgresql.org.
>
> Here there is mention of 'long writes and short writes', will trying out latest version of postgress help?

Well in the bug report the issue seems to be with BLCKSZ and XLOG_BLCKSZ.

So have you changed these?

I see the commit in my 9.3.5 sources, so if the above is indeed the
issue then using the latest version would help.

If that does not work then more information is needed.

What file system, OS, architecture(32/64bit) are you using?

What are the files you refer to below and how are they being
created/inserted?

>
> 4. I tried with 'archive_command = /usr/bin.true' but still I face the ""pg_xlog/xlogtemp.7884": No space left on
device"issue. 
>
> 5. Once I  hit ""pg_xlog/xlogtemp.7884": No space left on device" I start seeing it for other files as well:
>
>      "WARNING:  could not create relation-cache initialization file "base/16384/pg_internal.init.13715": No space
left
>       on device DETAIL:  Continuing anyway, but there's something wrong."
>
> 6. I am seeing this issue when I create around 60,000 to 70,000 files (each of size 4k) in the data base.
>
>
> Appreciate any quick help on this.
>
> --
> Thanks and regards,
> Rupesh



--
Adrian Klaver
adrian.klaver@aklaver.com

Adrian Klaver <adrian.klaver@aklaver.com> writes:
> On 09/28/2014 05:52 AM, Roopeshakumar Narayansa Shalgar (rshalgar) wrote:
>> 4. I tried with 'archive_command = /usr/bin.true' but still I face the ""pg_xlog/xlogtemp.7884": No space left on
device"issue. 

This error message is pretty specific: the OS is telling us we can't have
any more disk space.  If you're sure you're not out of disk space, then
the likely explanation is there's some sort of disk quota in effect.

            regards, tom lane

Roopeshakumar Narayansa Shalgar (rshalgar) wrote:
> Hi,
>
>
>
> I am using version 9.3.1 and see the “no space device error” even though there
> is enough space (99% free) on my disk.

Just to be sure, check the output of both 'df -h' (for disk blocks) and 'df -hi'
(for inodes). You might have ran out of inodes...

Eric