After that I moved the content of /data2/pgsql/baseb to actual datadir and tried to start up the cluster. On some servers it worked on the three didn't. The servers are of the same HW config and the prerequisites are on all servers set by puppet.
I'd like to ask for help clarifying an issue I'm having.
I've recently prepared new servers in another datacenter for some of our databases which I want to set up as a streaming replicas. There are several instances(clusters) with size ranging from 50-150GB. Some of them were set up with no issue. Three of them failed to start after pg_basebackup completed with:
FATAL: could not access status of transaction 923709700 DETAIL: Could not read from file "pg_clog/0370" at offset 237568: Success.
(the clog file differed in each case of course..)
As for PG versions one is 9.1.14 (on both master and replica) and the other two 9.2.9 (also on both)
I've checked each file in question on the master server and in each case it was last changed during the run of the pg_basebackup. I've copied the file from master to the slave and it started normally, succesfully connected to primary and everything seems ok since then.
My question is, why this is happening? Is it because the clog file was copied by pg_basebackup before the trasaction status had been written to it? Couldn't this information be recovered from xlog files? Is my current solution safe for the database (I suppose so, but I'd rather have it confirmed :) )
I'm puzzled as I haven't found this issue discussed at all so far and it happened to me three times in one day on three different servers..
Not sure what is going on, but it would help to see the pg_basebackup command issued.
