Thread: PostgreSQL benchmarked on XFS vs ZFS vs btrfs vs ext4
Hi,
Some months ago, I ran some (probably naive) benchmarks looking at how
pgbench performed on an identical system with differing filesystems. (on
Linux).
Since then the kernel-level version of ZFS became usable, and there have
been improvements to btrfs, and no doubt various updates in the Linux
kernel and PostgreSQL that should help performance.
I ran the tests on Ubuntu 11.04 with Pg 9.0 first, then upgraded the
system to Ubuntu 11.10 (beta) with Pg 9.1 and ran them again.
The latter combination showed a considerable performance improvement
overall - although I didn't investigate to find out whether this was due
to kernel improvements, postgres improvements, or virtio improvements.
The results are measured in transactions-per-second, with higher numbers
being better.
Results:
ext4 (data=writeback,relatime):
natty: 248
oneiric: 297
ext4 (data=writeback,relatime,nobarrier):
natty: didn't test
oneiric: 1409
XFS (relatime):
natty: didn't test
oneiric: 171
btrfs (relatime):
natty: 61.5
oneiric: 91
btrfs (relatime,nodatacow):
natty: didn't test
oneiric: 128
ZFS (defaults):
natty: 171
oneiric: 996
Conclusion:
Last time I ran these tests, xfs and ext4 pulled very similar results,
and both were miles ahead of btrfs. This time around, ext4 has managed
to get a significantly faster result than xfs.
However we have a new contender - ZFS performed *extremely* well on the
latest Ubuntu setup - achieving triple the performance of regular ext4!
I'm not sure how it achieved this, and whether we're losing some kind of
data protection (eg. like the "barrier" options in XFS and ext4).
If ext4 has barriers disabled, it surpasses even ZFSs high score.
Oddly, ZFS performed wildly differently on ubuntu 11.04 vs 11.10b. I
can't explain this. Any ideas?
Cheers,
Toby
On 09/13/2011 08:15 PM, Toby Corkindale wrote: > Hi, > Some months ago, I ran some (probably naive) benchmarks looking at how pgbench performed on an identical system with differingfilesystems. (on Linux). > > Since then the kernel-level version of ZFS became usable, and there have been improvements to btrfs, and no doubt variousupdates in the Linux kernel and PostgreSQL that should help performance. > > I ran the tests on Ubuntu 11.04 with Pg 9.0 first, then upgraded the system to Ubuntu 11.10 (beta) with Pg 9.1 and ranthem again. > > The latter combination showed a considerable performance improvement overall - although I didn't investigate to find outwhether this was due to kernel improvements, postgres improvements, or virtio improvements. > > The results are measured in transactions-per-second, with higher numbers being better. > > Results: > > ext4 (data=writeback,relatime): > natty: 248 > oneiric: 297 > > ext4 (data=writeback,relatime,nobarrier): > natty: didn't test > oneiric: 1409 > > XFS (relatime): > natty: didn't test > oneiric: 171 > > btrfs (relatime): > natty: 61.5 > oneiric: 91 > > btrfs (relatime,nodatacow): > natty: didn't test > oneiric: 128 > > ZFS (defaults): > natty: 171 > oneiric: 996 > > > Conclusion: > Last time I ran these tests, xfs and ext4 pulled very similar results, and both were miles ahead of btrfs. This time around,ext4 has managed to get a significantly faster result than xfs. > > However we have a new contender - ZFS performed *extremely* well on the latest Ubuntu setup - achieving triple the performanceof regular ext4! > I'm not sure how it achieved this, and whether we're losing some kind of data protection (eg. like the "barrier" optionsin XFS and ext4). > If ext4 has barriers disabled, it surpasses even ZFSs high score. > > Oddly, ZFS performed wildly differently on ubuntu 11.04 vs 11.10b. I can't explain this. Any ideas? > > > Cheers, > Toby > Did you test unplugging the power cable in the middle of a test to see which would come back up? -Andy
On 14/09/11 12:56, Andy Colson wrote: > On 09/13/2011 08:15 PM, Toby Corkindale wrote: >> Hi, >> Some months ago, I ran some (probably naive) benchmarks looking at how >> pgbench performed on an identical system with differing filesystems. >> (on Linux). >> [snip] > > Did you test unplugging the power cable in the middle of a test to see > which would come back up? Heh, no, but it'd be interesting to see.. I wonder if turning a virtual machine off has the same effect?
On 14/09/2011 09:30, Toby Corkindale wrote: > On 14/09/11 12:56, Andy Colson wrote: >> On 09/13/2011 08:15 PM, Toby Corkindale wrote: >>> Hi, >>> Some months ago, I ran some (probably naive) benchmarks looking at how >>> pgbench performed on an identical system with differing filesystems. >>> (on Linux). >>> > [snip] >> >> Did you test unplugging the power cable in the middle of a test to see >> which would come back up? > > Heh, no, but it'd be interesting to see.. > > I wonder if turning a virtual machine off has the same effect? No, never the exact same effect. There are two things to consider: if/when/how the OS flushes the data to the hardware and if/when/how the hardware flushes the data to physical storage. You can simulate only the failure of the first part with a virtual machine, but not the second (unless you bring down the VM host...).
Can you go into some more detail on how you set up ZFS on these systems? Robert Treat conjecture: xzilla.net consulting: omniti.com On Tue, Sep 13, 2011 at 10:56 PM, Andy Colson <andy@squeakycode.net> wrote: > On 09/13/2011 08:15 PM, Toby Corkindale wrote: >> >> Hi, >> Some months ago, I ran some (probably naive) benchmarks looking at how >> pgbench performed on an identical system with differing filesystems. (on >> Linux). >> >> Since then the kernel-level version of ZFS became usable, and there have >> been improvements to btrfs, and no doubt various updates in the Linux kernel >> and PostgreSQL that should help performance. >> >> I ran the tests on Ubuntu 11.04 with Pg 9.0 first, then upgraded the >> system to Ubuntu 11.10 (beta) with Pg 9.1 and ran them again. >> >> The latter combination showed a considerable performance improvement >> overall - although I didn't investigate to find out whether this was due to >> kernel improvements, postgres improvements, or virtio improvements. >> >> The results are measured in transactions-per-second, with higher numbers >> being better. >> >> Results: >> >> ext4 (data=writeback,relatime): >> natty: 248 >> oneiric: 297 >> >> ext4 (data=writeback,relatime,nobarrier): >> natty: didn't test >> oneiric: 1409 >> >> XFS (relatime): >> natty: didn't test >> oneiric: 171 >> >> btrfs (relatime): >> natty: 61.5 >> oneiric: 91 >> >> btrfs (relatime,nodatacow): >> natty: didn't test >> oneiric: 128 >> >> ZFS (defaults): >> natty: 171 >> oneiric: 996 >> >> >> Conclusion: >> Last time I ran these tests, xfs and ext4 pulled very similar results, and >> both were miles ahead of btrfs. This time around, ext4 has managed to get a >> significantly faster result than xfs. >> >> However we have a new contender - ZFS performed *extremely* well on the >> latest Ubuntu setup - achieving triple the performance of regular ext4! >> I'm not sure how it achieved this, and whether we're losing some kind of >> data protection (eg. like the "barrier" options in XFS and ext4). >> If ext4 has barriers disabled, it surpasses even ZFSs high score. >> >> Oddly, ZFS performed wildly differently on ubuntu 11.04 vs 11.10b. I can't >> explain this. Any ideas? >> >> >> Cheers, >> Toby >> > > Did you test unplugging the power cable in the middle of a test to see which > would come back up? > > -Andy > > -- > Sent via pgsql-general mailing list (pgsql-general@postgresql.org) > To make changes to your subscription: > http://www.postgresql.org/mailpref/pgsql-general >
Dne 14.9.2011 03:15, Toby Corkindale napsal(a): > Hi, > Some months ago, I ran some (probably naive) benchmarks looking at how > pgbench performed on an identical system with differing filesystems. (on > Linux). > > Since then the kernel-level version of ZFS became usable, and there have > been improvements to btrfs, and no doubt various updates in the Linux > kernel and PostgreSQL that should help performance. > > I ran the tests on Ubuntu 11.04 with Pg 9.0 first, then upgraded the > system to Ubuntu 11.10 (beta) with Pg 9.1 and ran them again. > > The latter combination showed a considerable performance improvement > overall - although I didn't investigate to find out whether this was due > to kernel improvements, postgres improvements, or virtio improvements. > > The results are measured in transactions-per-second, with higher numbers > being better. > > Results: > > ext4 (data=writeback,relatime): > natty: 248 > oneiric: 297 > > ext4 (data=writeback,relatime,nobarrier): > natty: didn't test > oneiric: 1409 > > XFS (relatime): > natty: didn't test > oneiric: 171 > > btrfs (relatime): > natty: 61.5 > oneiric: 91 > > btrfs (relatime,nodatacow): > natty: didn't test > oneiric: 128 > > ZFS (defaults): > natty: 171 > oneiric: 996 > > > Conclusion: > Last time I ran these tests, xfs and ext4 pulled very similar results, > and both were miles ahead of btrfs. This time around, ext4 has managed > to get a significantly faster result than xfs. > > However we have a new contender - ZFS performed *extremely* well on the > latest Ubuntu setup - achieving triple the performance of regular ext4! > I'm not sure how it achieved this, and whether we're losing some kind of > data protection (eg. like the "barrier" options in XFS and ext4). > If ext4 has barriers disabled, it surpasses even ZFSs high score. > > Oddly, ZFS performed wildly differently on ubuntu 11.04 vs 11.10b. I > can't explain this. Any ideas? Those results look very suspisicous I guess. Unless we know all the details about the fs options, pgbench runs and system (drives, cache, controller) it's very difficult to say anything. For example the ZFS results seem really strange - I've done a quite thorough benchmark of file systems last month (http://bit.ly/oEdkSH) and I really don't think the differences can be that huge under the same conditions - increase from 171 to 996 is ridiculous. The only explanation I can come up with is that oneiric uses significantly weaker ZFS options (e.g. disabled write barriers) and there's a controller with write cache. But that also means you should not compare ZFS to plain ext4 (297 tps) but to ext4 with write barriers disabled (1409 tps). That changes the results a bit, because instead of being 3x faster, ZFS is about 30% slower. Tomas
On 15/09/11 02:46, Robert Treat wrote: > Can you go into some more detail on how you set up ZFS on these systems? I'm afraid my knowledge of ZFS is rather weak compared to the other filesystems - all I really did was zpool create followed by zfs create, using all the defaults. The zpool was created against an LVM logical volume (which was the same one used for all the filesystems measured in the tests). That LV was itself part of a volume group that was striped over three disks (Western Digital WD1003FBYX). I'm happy to re-run the benchmark with different ZFS options if you can suggest some. Unfortunately I can't easily bypass the LVM layer here, as the disks involved are fully committed to the volume group and I don't want to rebuild the machine. Cheers, Toby > On Tue, Sep 13, 2011 at 10:56 PM, Andy Colson<andy@squeakycode.net> wrote: >> On 09/13/2011 08:15 PM, Toby Corkindale wrote: >>> >>> Hi, >>> Some months ago, I ran some (probably naive) benchmarks looking at how >>> pgbench performed on an identical system with differing filesystems. (on >>> Linux). >>> >>> Since then the kernel-level version of ZFS became usable, and there have >>> been improvements to btrfs, and no doubt various updates in the Linux kernel >>> and PostgreSQL that should help performance. >>> >>> I ran the tests on Ubuntu 11.04 with Pg 9.0 first, then upgraded the >>> system to Ubuntu 11.10 (beta) with Pg 9.1 and ran them again. >>> >>> The latter combination showed a considerable performance improvement >>> overall - although I didn't investigate to find out whether this was due to >>> kernel improvements, postgres improvements, or virtio improvements. >>> >>> The results are measured in transactions-per-second, with higher numbers >>> being better. >>> >>> Results: >>> >>> ext4 (data=writeback,relatime): >>> natty: 248 >>> oneiric: 297 >>> >>> ext4 (data=writeback,relatime,nobarrier): >>> natty: didn't test >>> oneiric: 1409 >>> >>> XFS (relatime): >>> natty: didn't test >>> oneiric: 171 >>> >>> btrfs (relatime): >>> natty: 61.5 >>> oneiric: 91 >>> >>> btrfs (relatime,nodatacow): >>> natty: didn't test >>> oneiric: 128 >>> >>> ZFS (defaults): >>> natty: 171 >>> oneiric: 996 >>> >>> >>> Conclusion: >>> Last time I ran these tests, xfs and ext4 pulled very similar results, and >>> both were miles ahead of btrfs. This time around, ext4 has managed to get a >>> significantly faster result than xfs. >>> >>> However we have a new contender - ZFS performed *extremely* well on the >>> latest Ubuntu setup - achieving triple the performance of regular ext4! >>> I'm not sure how it achieved this, and whether we're losing some kind of >>> data protection (eg. like the "barrier" options in XFS and ext4). >>> If ext4 has barriers disabled, it surpasses even ZFSs high score. >>> >>> Oddly, ZFS performed wildly differently on ubuntu 11.04 vs 11.10b. I can't >>> explain this. Any ideas?
On Tue, Sep 13, 2011 at 9:15 PM, Toby Corkindale <toby.corkindale@strategicdata.com.au> wrote: > However we have a new contender - ZFS performed *extremely* well on the > latest Ubuntu setup - achieving triple the performance of regular ext4! Did you do any tuning to ZFS? There are many tweaks to it, like putting a cache disk in front of it, or moving the logs to SSD and such. I haven't run any produciton DBs on ZFS yet, but it sure is tempting. The speed penalty for the features it gives you (snapshots, robust against power fails, etc.) is worth the tradeoff.
On Wed, Sep 14, 2011 at 11:18 PM, Toby Corkindale <toby.corkindale@strategicdata.com.au> wrote: > The zpool was created against an LVM logical volume (which was the same one > used for all the filesystems measured in the tests). That LV was itself part > of a volume group that was striped over three disks (Western Digital > WD1003FBYX). So you're throwing in the complication of a logical volume manager interfering with zfs. If you put ZFS on three separate drives, it has a better chance of optimizing its operations.
On 17/09/11 00:09, Vick Khera wrote: > On Tue, Sep 13, 2011 at 9:15 PM, Toby Corkindale > <toby.corkindale@strategicdata.com.au> wrote: >> However we have a new contender - ZFS performed *extremely* well on the >> latest Ubuntu setup - achieving triple the performance of regular ext4! > > Did you do any tuning to ZFS? There are many tweaks to it, like > putting a cache disk in front of it, or moving the logs to SSD and > such. I haven't run any produciton DBs on ZFS yet, but it sure is > tempting. The speed penalty for the features it gives you (snapshots, > robust against power fails, etc.) is worth the tradeoff. No, I didn't do that kind of tuning - agreed, it'd improve the performance. But then putting an SSD in the mix and storing journals on it would have improved the performance of XFS and ext4 as well.. I'll re-run the tests again in the future, no doubt, and hopefully I'll have a spare SSD by then. Also maybe I'll have learnt more about ZFS; I'm a bit of a noob at the moment. I agree that ZFS does seem to offer some rather nice features though! I'm tempted to start using it on my personal server now; although I'll be leaving it for some time before considering using it in production at work. Cheers, Toby