Speaking to the sys admins we found that the file system where the WAL was written to did not have direct_io and hence all writes were getting buffered. So we enabled direct_io on that FS and now writes are order of magnitude slower (5x times).
It seems PG already uses O_DIRECT for WAL. So is the enabling of direct_io in the mount options of the FS totally redundant. It definitely has a serious performance impact.
Thanks
From: srkrishna@outlook.com To: pgsql-admin@postgresql.org Subject: [ADMIN] Best setting for wal_sync_method Date: Thu, 7 May 2015 09:36:36 -0400
On RHEL 6.4 which is the best setting for wal_sync_method for write intensive operations. Is the default fdatasync good for all conditions, including reliability.