Question: BlockSize > 8192 with FusionIO - Mailing list pgsql-performance

From Strange, John W
Subject Question: BlockSize > 8192 with FusionIO
Date
Msg-id EF37296944B47C40ADDCCB7BFD6289FE046EED92AB@EMASC201VS01.exchad.jpmchase.net
Whole thread Raw
In response to Re: CPU bound  (Mladen Gogala <mladen.gogala@vmsinfo.com>)
Responses Re: Question: BlockSize > 8192 with FusionIO  (Greg Smith <greg@2ndquadrant.com>)
List pgsql-performance
Has anyone had a chance to recompile and try larger a larger blocksize than 8192 with pSQL 8.4.x?  I'm finally getting
aroundto tuning some FusionIO drives that we are setting up.  We are looking to setup 4 fusionIO drives per server, and
thenuse pgpooler to scale them to 3 servers so that we can scale up to 72 processors.  I'm almost done with the
configurationto start heavily testing but wanted to know if anyone has really messed with the blocksize options?
 

If anyone has done load balancing with pgpooler I'd love to hear their experience with it as well.  I have attached the
randwriteperformance test below, as you can see going from 8K -> 32K -> 1M blocksize the drives really start to move.
 

Thanks,
- John

[v025554@athenaash05 /]$ fio --filename=/fusionIO/export1/test1 --size=25G --bs=8k --direct=1 --rw=randwrite
--numjobs=8--runtime=30 --group_reporting --name=file1
 
file1: (g=0): rw=randwrite, bs=8K-8K/8K-8K, ioengine=sync, iodepth=1
...
file1: (g=0): rw=randwrite, bs=8K-8K/8K-8K, ioengine=sync, iodepth=1
Starting 8 processes
Jobs: 8 (f=8): [wwwwwwww] [100.0% done] [0K/138M /s] [0/17K iops] [eta 00m:00s]
file1: (groupid=0, jobs=8): err= 0: pid=23287
  write: io=3,819MB, bw=127MB/s, iops=16,292, runt= 30001msec
    clat (usec): min=41, max=1,835K, avg=268.42, stdev=3627.29
    bw (KB/s) : min=    4, max=142304, per=15.13%, avg=19714.31, stdev=8364.40
  cpu          : usr=0.16%, sys=3.13%, ctx=1123544, majf=0, minf=176
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued r/w: total=0/488779, short=0/0
     lat (usec): 50=14.81%, 100=58.17%, 250=0.28%, 500=22.33%, 750=3.67%
     lat (usec): 1000=0.16%
     lat (msec): 2=0.35%, 4=0.09%, 10=0.11%, 20=0.01%, 50=0.01%
     lat (msec): 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
     lat (msec): 2000=0.01%

Run status group 0 (all jobs):
  WRITE: io=3,819MB, aggrb=127MB/s, minb=130MB/s, maxb=130MB/s, mint=30001msec, maxt=30001msec

Disk stats (read/write):
  md0: ios=0/514993, merge=0/0, ticks=0/0, in_queue=0, util=0.00%, aggrios=0/0, aggrmerge=0/0, aggrticks=0/0,
aggrin_queue=0,aggrutil=0.00%
 
    fiod: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=nan%
    fioc: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=nan%
    fiob: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=nan%
    fioa: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=nan%
[v025554@athenaash05 /]$ fio --filename=/fusionIO/export1/test1 --size=25G --bs=32k --direct=1 --rw=randwrite
--numjobs=8--runtime=30 --group_reporting --name=file1
 
file1: (g=0): rw=randwrite, bs=32K-32K/32K-32K, ioengine=sync, iodepth=1
...
file1: (g=0): rw=randwrite, bs=32K-32K/32K-32K, ioengine=sync, iodepth=1
Starting 8 processes
Jobs: 8 (f=8): [wwwwwwww] [100.0% done] [0K/343M /s] [0/11K iops] [eta 00m:00s]
file1: (groupid=0, jobs=8): err= 0: pid=23835
  write: io=9,833MB, bw=328MB/s, iops=10,487, runt= 30002msec
    clat (usec): min=64, max=227K, avg=349.31, stdev=1517.64
    bw (KB/s) : min=  883, max=171712, per=16.25%, avg=54548.49, stdev=13973.76
  cpu          : usr=0.18%, sys=2.82%, ctx=669138, majf=0, minf=176
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued r/w: total=0/314659, short=0/0
     lat (usec): 100=84.14%, 250=8.73%, 500=0.07%, 750=2.45%, 1000=3.19%
     lat (msec): 2=0.29%, 4=0.17%, 10=0.22%, 20=0.23%, 50=0.31%
     lat (msec): 100=0.13%, 250=0.05%

Run status group 0 (all jobs):
  WRITE: io=9,833MB, aggrb=328MB/s, minb=336MB/s, maxb=336MB/s, mint=30002msec, maxt=30002msec

Disk stats (read/write):
  md0: ios=0/455522, merge=0/0, ticks=0/0, in_queue=0, util=0.00%, aggrios=0/0, aggrmerge=0/0, aggrticks=0/0,
aggrin_queue=0,aggrutil=0.00%
 
    fiod: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=nan%
    fioc: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=nan%
    fiob: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=nan%
    fioa: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=nan%
[v025554@athenaash05 /]$ fio --filename=/fusionIO/export1/test1 --size=25G --bs=1M --direct=1 --rw=randwrite
--numjobs=8--runtime=30 --group_reporting --name=file1
 
file1: (g=0): rw=randwrite, bs=1M-1M/1M-1M, ioengine=sync, iodepth=1
...
file1: (g=0): rw=randwrite, bs=1M-1M/1M-1M, ioengine=sync, iodepth=1
Starting 8 processes
Jobs: 8 (f=8): [wwwwwwww] [100.0% done] [0K/912M /s] [0/890 iops] [eta 00m:00s]
file1: (groupid=0, jobs=8): err= 0: pid=24877
  write: io=25,860MB, bw=862MB/s, iops=861, runt= 30004msec
    clat (usec): min=456, max=83,766, avg=5599.02, stdev=2026.74
    bw (KB/s) : min=28603, max=216966, per=11.93%, avg=105311.22, stdev=10668.06
  cpu          : usr=0.06%, sys=2.03%, ctx=91888, majf=0, minf=176
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued r/w: total=0/25860, short=0/0
     lat (usec): 500=12.74%, 750=20.60%, 1000=7.37%
     lat (msec): 2=3.12%, 4=6.57%, 10=26.95%, 20=21.37%, 50=1.12%
     lat (msec): 100=0.14%

Run status group 0 (all jobs):
  WRITE: io=25,860MB, aggrb=862MB/s, minb=883MB/s, maxb=883MB/s, mint=30004msec, maxt=30004msec

Disk stats (read/write):
  md0: ios=0/500382, merge=0/0, ticks=0/0, in_queue=0, util=0.00%, aggrios=0/0, aggrmerge=0/0, aggrticks=0/0,
aggrin_queue=0,aggrutil=0.00%
 
    fiod: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=nan%
    fioc: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=nan%
    fiob: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=nan%
    fioa: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=nan%

This communication is for informational purposes only. It is not
intended as an offer or solicitation for the purchase or sale of
any financial instrument or as an official confirmation of any
transaction. All market prices, data and other information are not
warranted as to completeness or accuracy and are subject to change
without notice. Any comments or statements made herein do not
necessarily reflect those of JPMorgan Chase & Co., its subsidiaries
and affiliates.

This transmission may contain information that is privileged,
confidential, legally privileged, and/or exempt from disclosure
under applicable law. If you are not the intended recipient, you
are hereby notified that any disclosure, copying, distribution, or
use of the information contained herein (including any reliance
thereon) is STRICTLY PROHIBITED. Although this transmission and any
attachments are believed to be free of any virus or other defect
that might affect any computer system into which it is received and
opened, it is the responsibility of the recipient to ensure that it
is virus free and no responsibility is accepted by JPMorgan Chase &
Co., its subsidiaries and affiliates, as applicable, for any loss
or damage arising in any way from its use. If you received this
transmission in error, please immediately contact the sender and
destroy the material in its entirety, whether in electronic or hard
copy format. Thank you.

Please refer to http://www.jpmorgan.com/pages/disclosures for
disclosures relating to European legal entities.

pgsql-performance by date:

Previous
From: Mladen Gogala
Date:
Subject: Re: CPU bound
Next
From: Greg Smith
Date:
Subject: Re: Question: BlockSize > 8192 with FusionIO