On Wed, 16 Oct 2024 17:06:24 +0300
Onni Hakala <onni@fyff.ee> wrote:
> Hey,
Hi,
> I have a large dataset of > 100TB which would be very expensive to
> store solely into SSD drives.
>
> I have access to a server which has 2x 3.84TB NVME SSD disks and
> large array of HDD drives 8 x 22TB.
>
> Most of the data that I have in my dataset is very rarely accessed
> and is stored only for archival purposes.
I can't answer to the rest but with the configuration you describe,
you might meet some slow down because HDD speed is not what you think
it is when you reach a barrier in disks filling.
A test on a 2TB/7200RPM returns that, at the beginning of the disk it is
quite fast :
========================================================================
hdparm --offset 1g --direct -t /dev/sdb1
/dev/sdb1:
geometry = 243201/255/63, sectors = 3907027087, start = 2048
Timing O_DIRECT disk reads (offset 1 GB): 504 MB in 3.00 seconds =
167.97 MB/sec
@ 50%, there's even a small gain (NB : this is a PITA, hdparm displays
GB where it uses in fact GiB:( :
=======================================================================
hdparm --offset 931g --direct -t /dev/sdb1
/dev/sdb1:
geometry = 243201/255/63, sectors = 3907027087, start = 2048
Timing O_DIRECT disk reads (offset 931 GB): 520 MB in 3.01 seconds =
172.73 MB/sec
@ 70% of the disk, the loss is only 11% :
=========================================
hdparm --offset 1304g --direct -t /dev/sdb1
/dev/sdb1:
geometry = 243201/255/63, sectors = 3907027087, start = 2048
Timing O_DIRECT disk reads (offset 1304 GB): 440 MB in 3.01 seconds =
146.38 MB/sec
@ 80% of the disk the read speed has already lost 21% :
=======================================================
hdparm --offset 1490g --direct -t /dev/sdb1
/dev/sdb1:
geometry = 243201/255/63, sectors = 3907027087, start = 2048
Timing O_DIRECT disk reads (offset 1490 GB): 390 MB in 3.01 seconds =
129.68 MB/sec
but @ 90% of the disk, the loss climbs @ 30.5% :
================================================
hdparm --offset 1676g --direct -t /dev/sdb1
/dev/sdb1:
geometry = 243201/255/63, sectors = 3907027087, start = 2048
Timing O_DIRECT disk reads (offset 1676 GB): 344 MB in 3.01 seconds =
114.45 MB/sec
So, for a good response time, it is best to never fill your disks
further than 75% and it is also always a good practice to avoid
some inconvenience, especially when using a COW FS, such as ZFS.
Jean-Yves
--
