Replication, slave becomes master, pgbouncer restarts with new config, everyone happy.
Until soon we noticed strange errors. Turns out data was broken somewhere - we had unique indexes with duplicate records, we had missing chunks in toast tables, and the like.
Scale of the problems wasn't clear and I fixed some of the most obvious problems - recreated indices, cleared duplicates. Around 0:30 I started database-wide VACUUM, it started printing messages about fixing things. Everything seemed to work normally.
Around 3:30 database stopped responding to queries; it was noticed around 9:30 and I attempted to stop VACUUM (ctrl+c in psql command line). 30 minutes and nothing. I decided to restart postmaster. After another 30 minutes we switched back to old server. 3 hours passed and new server postmaster is still shutting down. And I would like to get it back to restore yesterday data.
At this point I could do with new server anything that will get database up and running sooner. kill -9, start, wait for recovery? Server reboot?
