Re: Jira database won't start after disk filled up - Mailing list pgsql-general

From Vick Khera
Subject Re: Jira database won't start after disk filled up
Date
Msg-id CALd+dcfdfWmnOqdjpK8P1920SM8wG4nyAm7y=T_bcx+D1Z7Bdw@mail.gmail.com
Whole thread Raw
In response to Jira database won't start after disk filled up  (Paul Costello <paulc1217@gmail.com>)
Responses Re: Jira database won't start after disk filled up  (Paul Costello <paulc1217@gmail.com>)
List pgsql-general
On Fri, Mar 2, 2018 at 4:32 PM, Paul Costello <paulc1217@gmail.com> wrote:
I have a database that wouldn't start due to the disk filling up back on 1/10, unbeknownst to us until 2/27.  This is jira, so it's critical data.  It appears jira was running in memory that entire time.


Those first two sentences seem contradictory...
 

I needed to run pg_resetxlog -f in order to start the database.  It started, but upon logging in I found the system catalog and some data to be corrupt. 


Once you did this, fixing the data is really on you. Postgres has no way to know what any of the data mean, nor how to decide what to keep and what to toss on those conflicting rows with duplicate keys.

What I'd personally do is take your 1/5 backup, then merge in rows for tickets and affiliated data from whatever you can recover in the current database copy you have. Once that's done, run jira's built-in integrity checker then do a full export to XML backup format. Finally re-import that into a fresh jira so you know what's in there is consistent.  You'll probably also have to cross-reference the attachments directory for missing tickets and clean up those files (or synthesize tickets for them).

If your jira is configured to send email somewhere on ticket updates, gathering those (even if it is in multiple people's mailboxes) and recreating ticket info from them would also move you along.

You will lose some of your data because not all of it was written to disk.

pgsql-general by date:

Previous
From: Vick Khera
Date:
Subject: Re: Is there a continuous backup for pg ?
Next
From: Paul Costello
Date:
Subject: Re: Jira database won't start after disk filled up