Thread: PostgreSQL with BDR - PANIC: could not create replication identifier checkpoint

This e-mail and any attachments are intended only for use by the addressee(s) named herein and may contain confidential information. If you are not the intended recipient of this e-mail, you are hereby notified any dissemination, distribution or copying of this email and any attachments is strictly prohibited. If you receive this email in error, please immediately notify the sender by return email and permanently delete the original, any copy and any printout thereof. The integrity and security of e-mail cannot be guaranteed.

Cameron Smith wrote:

> t:2016-05-19 01:14:51.668 UTC d= p=144 a=PANIC:  could not create replication identifier checkpoint
"pg_logical/checkpoints/8-F3923F98.ckpt.tmp":Invalid argument 

This line corresponds to the following code in BDR's 9.4.4
src/backend/replication/logical/replication_identifier.c:

    /*
     * no other backend can perform this at the same time, we're protected by
     * CheckpointLock.
     */
    tmpfd = OpenTransientFile(tmppath,
                              O_CREAT | O_EXCL | O_WRONLY | PG_BINARY,
                              S_IRUSR | S_IWUSR);
    if (tmpfd < 0)
        ereport(PANIC,
                (errcode_for_file_access(),
                 errmsg("could not create replication identifier checkpoint \"%s\": %m",
                        tmppath)));

This file does not exist in 9.5, but instead we have
src/backend/replication/logical/origin.c which has identical code.

OpenTransientFile calls BasicOpenFile, which in turn calls open() and
propagates the errno.  My manpage doesn't list any possible reasons for
open() to return EINVAL, so I'm at a loss about what is happening here.
Maybe this is a filesystem problem?

--
Álvaro Herrera                http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

## Cameron Smith (csmith@stereodllc.com):

> t:2016-05-19 01:14:51.668 UTC d= p=144 a=PANIC:  could not create replication identifier checkpoint
"pg_logical/checkpoints/8-F3923F98.ckpt.tmp":Invalid argument 
> t:2016-05-19 01:14:51.671 UTC d= p=9729 a=WARNING:  could not create relation-cache initialization file
"global/pg_internal.init.9729":Invalid argument 

This and the other "Invalid Argument" errors give the impression that
your "power issue" has wrecked your filesystem - as Alvaro already
pointed out, there's no sane way that an open() returns EINVAL in
PostgreSQL on any "recommended" filesystem.
This shouldn't happen with any decently modern filesystem and "safe"
hardware. Did you tweak anything on your filesystem? Did you check
your storage stack against https://wiki.postgresql.org/wiki/Reliable_Writes?

Regards,
Christoph

--
Spare Space

I'd agree:  most likely a file system problem.  Is there any hope that this file could be re-built?

My current plan is to use bdr_part_by_node_names to remove the failing node and then rebuild it from a fresh backup
(andprobably on a new server). 

Thank you for your help!

Cameron Smith


________________________________________
From: Alvaro Herrera <alvherre@2ndquadrant.com>
Sent: May 19, 2016 2:56 PM
To: Cameron Smith
Cc: pgsql-general@postgresql.org
Subject: Re: [GENERAL] PostgreSQL with BDR - PANIC:  could not create replication identifier checkpoint

CAUTION EXTERNAL EMAIL






Cameron Smith wrote:

> t:2016-05-19 01:14:51.668 UTC d= p=144 a=PANIC:  could not create replication identifier checkpoint
"pg_logical/checkpoints/8-F3923F98.ckpt.tmp":Invalid argument 

This line corresponds to the following code in BDR's 9.4.4
src/backend/replication/logical/replication_identifier.c:

    /*
     * no other backend can perform this at the same time, we're protected by
     * CheckpointLock.
     */
    tmpfd = OpenTransientFile(tmppath,
                              O_CREAT | O_EXCL | O_WRONLY | PG_BINARY,
                              S_IRUSR | S_IWUSR);
    if (tmpfd < 0)
        ereport(PANIC,
                (errcode_for_file_access(),
                 errmsg("could not create replication identifier checkpoint \"%s\": %m",
                        tmppath)));

This file does not exist in 9.5, but instead we have
src/backend/replication/logical/origin.c which has identical code.

OpenTransientFile calls BasicOpenFile, which in turn calls open() and
propagates the errno.  My manpage doesn't list any possible reasons for
open() to return EINVAL, so I'm at a loss about what is happening here.
Maybe this is a filesystem problem?

--
Álvaro Herrera                http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
DO NOT open attachments or click on links from unknown senders or unexpected emails





This e-mail and any attachments are intended only for use by the addressee(s) named herein and may contain confidential
information.If you are not the intended recipient of this e-mail, you are hereby notified any dissemination,
distributionor copying of this email and any attachments is strictly prohibited. If you receive this email in error,
pleaseimmediately notify the sender by return email and permanently delete the original, any copy and any printout
thereof.The integrity and security of e-mail cannot be guaranteed. 

El 19/05/16 a las 16:15, Cameron Smith escribió:
> I'd agree:  most likely a file system problem.  Is there any hope that this file could be re-built?
>
> My current plan is to use bdr_part_by_node_names to remove the failing node and then rebuild it from a fresh backup
(andprobably on a new server). 

I think the most sensible plan is to remove the node from the bdr
cluster with bdr_part_by_node_name(), maybe clean up the bdr_nodes table
(some won't be happy with me suggesting this :)), remove the data
directory on the failed node and rejoin with bdr_init_copy

I'd suggest following the suggestions from Christoph and check that you
have a sane file-system configuration.

Also check if you didn't end up with a damaged disk (run some stress
test on the hardware).

If this is on production (not a toy installation) I would suggest
replacing the disks all together.

Regards,

--
Martín Marqués                http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services