Yes, contradictory. We recently disabled email forwarding, so on 1/10 when the disk filled up, we never received any alerts. New position, so I was completely unaware of this database, except on a conceptual level, until Tuesday.
I think the best I can do is get this database back to 1/10. In my first restore attempt I noticed that the most recent jiraissue.resolutiondate was 1/10, so jira was running in memory the entire time and nothing was flushed to the db - probably because it was corrupted, and possibly other reasons related to neglect.
My hope is that I can get the db back to 1/10 and maybe we can, with Atlassian's help, somehow sync the lucene files back to the db. I don't think I will have any postgres data to work with beyond 1/10.
Does this still sound do-able with that kind of data gap?
On Fri, Mar 2, 2018 at 3:44 PM, Vick Khera <vivek@khera.org> wrote:
I have a database that wouldn't start due to the disk filling up back on 1/10, unbeknownst to us until 2/27. This is jira, so it's critical data. It appears jira was running in memory that entire time.
Those first two sentences seem contradictory...
I needed to run pg_resetxlog -f in order to start the database. It started, but upon logging in I found the system catalog and some data to be corrupt.
Once you did this, fixing the data is really on you. Postgres has no way to know what any of the data mean, nor how to decide what to keep and what to toss on those conflicting rows with duplicate keys.
What I'd personally do is take your 1/5 backup, then merge in rows for tickets and affiliated data from whatever you can recover in the current database copy you have. Once that's done, run jira's built-in integrity checker then do a full export to XML backup format. Finally re-import that into a fresh jira so you know what's in there is consistent. You'll probably also have to cross-reference the attachments directory for missing tickets and clean up those files (or synthesize tickets for them).
If your jira is configured to send email somewhere on ticket updates, gathering those (even if it is in multiple people's mailboxes) and recreating ticket info from them would also move you along.
You will lose some of your data because not all of it was written to disk.
