warm standby replication safe failover - Mailing list pgsql-novice

I'm new here and have a question. I'm working on a replacement PostgreSQL 8.4 setup for a small company. They now have a Primary postgres server and 2 secondaries which are replicated using Slony-I, however the consulting company that set it up is afraid to failover and keeping Slony running is taking lots of resources.

The server processes IVR phone calls so it must be quick and have high availability.

I'm setting up a replacement group of physical servers that we can migrate the data over to piece by piece. Since it is a small company, they've agreed that Slony has been too complicated and that in an emergency failover 1 to 5 minutes of lost data is acceptable and quickly having another server available is more important. I can setup 3 separate servers at 3 separate physical locations parallel to the current production servers.

I'm testing a setup with a primary server, which has archiving turned on and rsync's over ssh the WAL log files to a secondary server. The secondary server is running pg_standby and waiting for a trigger. A third server will just accept nightly dumps of the primary to keep load down on the primary using a cronjob. The third server will be up to 1 day old with the dumps, but ready to go in case of primary and secondary failure as well as 1 day historical dumps in case we need any old data.

My question is how to safely failover between Primary and Secondary servers in this warm standby setup without loosing any data? I'd like to switch back and forth periodically to ensure we can trigger a failover in an emergency. Using the trigger file on the Secondary, we could loose up to 5 minutes of data since the WAL files are only rotated every 5 minutes, even though archive_timeout = 60. I saw a blog that mentioned copying the pg_xlog files from the Primary to the Secondary in addition to pg_standby, but I'm unsure if this will make sure we have 100% of our data. I tried this, put data in at each step and recovered all the data, but I'm unsure if the process ensures that no data is lost.

Test setup: Primary was shipping logs to Secondary and secondary was running pg_standby. I stop shipping logs by setting archive_command = '' and reloading postgres on Primary. Then I create a table with data that doesn't make it to the Secondary. On Primary I stopped postgres. Secondary was still waiting for the next log, never receiving the new table. I rsync'ed just the pg_xlog files using "pg_xlog/0*" to the Secondary pg_xlog directory (not the WAL archive directory pg_standby will recover from). These are the latest WAL log files and "000000010000000000000003.00000020.backup" (should I exclude backup file?). Then I triggered the recover. The new table is there and accessible.

My guess is that the logs in the WAL archive directory have been applied to the database and then when the database starts up it also has the last logs in the pg_xlog directory so it applies those last ones as well and we don't loose any data. Can someone confirm or deny this or a better solution to any of the above?

Thank you so much for your help,

Andrew

--
Andrew Puschak
(267) 614 - 2373
apuschak@gmail.com
http://andrewpuschak.com