Hello, our postgresql 9.2.4 qa database (thankfully its just qa) seems to be hosed.
Starting at around 3:39am last night I started seeing errors about missing files and now I cannot run a pgdump or a vacuum without it complaining about files that it cannot find with errors like this: ERROR: could not open file "base/125542/12631". When I check the filesystem the files are indeed not there. The 1am regular vacuum completed and its log is clean. The postgres log is clean before these errors occurred.
Since this is qa we do not perform backups, and the solution if we cannot repair the problem will be to create a fresh qa server but I am intrigued about how to determine the source of the problem and the extent of the problem.
Is there a way to force vacuum to continue on errors or an alternate way to help determine all the missing files?
It might be totally unrelated, but yesterday morning on this qa server I stopped postgres, and created a symlink to pg_xlog so that it was writing to a different volume, and restarted. This was working fine all day so its possibly a red herring but I thought I should mention it.
