Thread: EINTR causes panic (data dir on btrfs)
Every few weeks, I'm getting a error like this:
> 2015-02-11 15:31:00 CET PANIC: could not write to log file 00000001000000070000007D at offset 1335296, length 8192:
Interruptedsystem call
> 2015-02-11 15:31:00 CET STATEMENT: COMMIT
> 2015-02-11 15:31:17 CET LOG: server process (PID 8390) was terminated by signal 6: Aborted
> 2015-02-11 15:31:17 CET DETAIL: Failed process was running: COMMIT
> 2015-02-11 15:31:17 CET LOG: terminating any other active server processes
> 2015-02-11 15:31:17 CET WARNING: terminating connection because of crash of another server proces
I'm running the Ubuntu 9.3.4-1 package on a 3.2.13 kernel.
Is there any solution for this? The code generating the error seems to
be this:
> if (write(openLogFile, from, nbytes) != nbytes)
> {
> /* if write didn't set errno, assume no disk space */
> if (errno == 0)
> errno = ENOSPC;
> ereport(PANIC,
> (errcode_for_file_access(),
> errmsg("could not write to log file %s "
> "at offset %u, length %lu: %m",
> XLogFileNameP(ThisTimeLineID, openLogSegNo),
> openLogOff, (unsigned long) nbytes)));
> }
which strikes me as a bit strange (but there may be data consistency
issues I'm not aware of). Why wouldn't postgres retry on EINTR or even
allow return values of write() lower than nbytes (and then continue in a
loop).
--
Gustavo Lopes
Gustavo Lopes wrote:
> Every few weeks, I'm getting a error like this:
>
> > 2015-02-11 15:31:00 CET PANIC: could not write to log file 00000001000000070000007D at offset 1335296, length 8192:
Interruptedsystem call
> > 2015-02-11 15:31:00 CET STATEMENT: COMMIT
> > 2015-02-11 15:31:17 CET LOG: server process (PID 8390) was terminated by signal 6: Aborted
> > 2015-02-11 15:31:17 CET DETAIL: Failed process was running: COMMIT
> > 2015-02-11 15:31:17 CET LOG: terminating any other active server processes
> > 2015-02-11 15:31:17 CET WARNING: terminating connection because of crash of another server proces
>
> I'm running the Ubuntu 9.3.4-1 package on a 3.2.13 kernel.
>
> Is there any solution for this? The code generating the error seems to
> be this:
>
> > if (write(openLogFile, from, nbytes) != nbytes)
> > {
> > /* if write didn't set errno, assume no disk space */
> > if (errno == 0)
> > errno = ENOSPC;
> > ereport(PANIC,
> > (errcode_for_file_access(),
> > errmsg("could not write to log file %s "
> > "at offset %u, length %lu: %m",
> > XLogFileNameP(ThisTimeLineID, openLogSegNo),
> > openLogOff, (unsigned long) nbytes)));
> > }
>
> which strikes me as a bit strange (but there may be data consistency
> issues I'm not aware of). Why wouldn't postgres retry on EINTR or even
> allow return values of write() lower than nbytes (and then continue in a
> loop).
I happened to notice this report from 15 months ago, which didn't get
any response. Did you find a solution to this problem? I would first
blame btrfs, mostly because I've never heard of anyone with this problem
on more mainstream filesystems. As I recall, we use SA_RESTART almost
everywhere so we don't expect EINTR anywhere.
--
Álvaro Herrera http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
Alvaro Herrera <alvherre@2ndquadrant.com> writes:
> Gustavo Lopes wrote:
>> ... Why wouldn't postgres retry on EINTR or even
>> allow return values of write() lower than nbytes (and then continue in a
>> loop).
> I happened to notice this report from 15 months ago, which didn't get
> any response. Did you find a solution to this problem? I would first
> blame btrfs, mostly because I've never heard of anyone with this problem
> on more mainstream filesystems. As I recall, we use SA_RESTART almost
> everywhere so we don't expect EINTR anywhere.
On my RHEL6 box, the signal(7) man page specifically states that write()
to a disk file will not return EINTR if interrupted by a signal handler
with SA_RESTART set:
If a blocked call to one of the following interfaces is interrupted by
a signal handler, then the call will be automatically restarted after
the signal handler returns if the SA_RESTART flag was used; otherwise
the call will fail with the error EINTR:
* read(2), readv(2), write(2), writev(2), and ioctl(2) calls on
"slow" devices. A "slow" device is one where the I/O call may
block for an indefinite time, for example, a terminal, pipe, or
socket. (A disk is not a slow device according to this defini-
tion.) If an I/O call on a slow device has already transferred
some data by the time it is interrupted by a signal handler, then
the call will return a success status (normally, the number of
bytes transferred).
We set SA_RESTART on every signal that we set an interrupt handler for.
So it's hard to avoid the conclusion that btrfs has broken the API
contract specified by signal(7).
A bit of googling suggests that at least one such case used to exist
in btrfs but has been fixed:
http://linux-btrfs.vger.kernel.narkive.com/CbodH9VP/patch-btrfs-don-t-return-eintr
I wonder what kernel version the OP was using.
regards, tom lane
On 13-05-2016 19:04, Alvaro Herrera wrote: > > I happened to notice this report from 15 months ago, which didn't get > any response. Did you find a solution to this problem? I would first > blame btrfs, mostly because I've never heard of anyone with this problem > on more mainstream filesystems. As I recall, we use SA_RESTART almost > everywhere so we don't expect EINTR anywhere. > I have not seen this anymore, no, though I'm not running Linux 3.2 on any machine anymore. Indeed it was on a btrfs filesystem, so this probably was a btrfs bug after all. -- Gustavo Lopes