Re: ZFS filesystem - supported ? - Mailing list pgsql-general

From Robert L Mathews
Subject Re: ZFS filesystem - supported ?
Date
Msg-id 2b41a967-d1ff-7f71-436a-b54f1a0d9e5a@tigertech.com
Whole thread Raw
In response to Re: ZFS filesystem - supported ?  (Mladen Gogala <gogala.mladen@gmail.com>)
Responses Re: ZFS filesystem - supported ?
List pgsql-general
On 10/25/21 1:40 PM, Mladen Gogala wrote:
> This is probably not the place 
> to discuss the inner workings of snapshots, but it is worth knowing that 
> snapshots drastically increase the IO rate on the file system - for 
> every snapshot. That's where the slowness comes from.

I have recent anecdotal experience of this. I experiment with using 
Btrfs for a 32 TB backup system that has five 8 TB spinning disks. 
There's an average of 8 MBps of writes scattered around the disks, which 
isn't super high, obviously.

The results were vaguely acceptable until I created a snapshot of it, at 
which point it became completely unusable. Even having one snapshot 
present caused hundreds of btrfs-related kernel threads to thrash in the 
"D" state almost constantly, and it never stopped doing that even when 
left for many hours.

I then experimented with adding a bcache layer on top of Btrfs to see if 
it would help. I added a 2 TB SSD using bcache, partitioned as 1900 GB 
read cache and 100 GB write cache. It made very little difference and 
was still unusable as soon as a snapshot was taken.

I did play with the various btrfs and bcache tuning knobs quite a bit 
and couldn't improve it.

Since that test was a failure, I then decided to try the same setup with 
OpenZFS on a lark, with the same set of disks in a "raidz" array, with 
the 2 TB SSD as an l2arc read cache (no write cache). It easily handles 
the same load, even with 72 hourly snapshots present, with the default 
settings. I'm actually quite impressed with it.

I'm sure that the RAID, snapshots and copy-on-write reduce the maximum 
performance considerably, compared to ext4. But on the other hand, it 
did provide the performance I expected to be possible given the setup. 
Btrfs *definitely* didn't; I was surprised at how badly it performed.

-- 
Robert L Mathews, Tiger Technologies, http://www.tigertech.net/



pgsql-general by date:

Previous
From: Mladen Gogala
Date:
Subject: Re: ZFS filesystem - supported ?
Next
From: Benedict Holland
Date:
Subject: Re: ZFS filesystem - supported ?