Thread: High IOWAIT times, low iops? Can't figure out what the bottleneck is...
I was wondering if you guys have some suggested settings for our server, i think we are not hardware limited but the configureation is set up incorrectly. For some reason our database seems to have trouble handling 5-10+ inserts per second which seems to be a pretty trivial load for this hardware, we're seeing very high %iowait, this is a pretty typical output for #iostat -m 5
avg-cpu: %user %nice %system %iowait %steal %idle
0.97 0.00 0.82 89.69 0.00 8.52
Device: tps MB_read/s MB_wrtn/s MB_read MB_wrtn
sda 85.40 0.18 0.72 0 3
sdb 0.00 0.00 0.00 0 0
sdc 30.43 0.00 0.76 0 3
sdd 93.91 0.71 0.12 3 0
sda = 2x320GB 7200rpm in RAID1 (operating system)
sdc = 2x150GB 10krpm in RAID1 (transaction log is on this array)
sdd = 6x150GB 10krpm in RAID 10 (database is on the array)
we're using ReiserFS on each of the arrays as the file system
OS is debian Etch x64
raid controller = 3ware 9650 12port - 256MB cache
8GB RAM, core 2 duo - quad core 2.4ghz 8MB L2 Cache
it would seem like the io subsystem is the limiting factor, but i feel like we should be barely hitting a wall, you can see from the example its writing < 2MB/s to the array. If i try to copy a big file (1-2GB) over to the array, the array hits 300MB/s sustained, it can burst write files < 256mb AT 800MB/S
Here's some of our settings
shared_buffers = 1024MB
temp_buffers = 32MB
max_prepared_transactions = 50
work_mem = 64MB
maintenance_work_mem = 256MB
max_stack_depth = 7MB
max_fsm_pages = 512000
vacuum_cost_delay = 20ms
fsync = on
wal_buffers = 1MB
checkpoint_segments = 32
effective_cache_size = 4128MB
If you guys have any suggestions it would be greatly appreciated
-Evan Reiser
avg-cpu: %user %nice %system %iowait %steal %idle
0.97 0.00 0.82 89.69 0.00 8.52
Device: tps MB_read/s MB_wrtn/s MB_read MB_wrtn
sda 85.40 0.18 0.72 0 3
sdb 0.00 0.00 0.00 0 0
sdc 30.43 0.00 0.76 0 3
sdd 93.91 0.71 0.12 3 0
sda = 2x320GB 7200rpm in RAID1 (operating system)
sdc = 2x150GB 10krpm in RAID1 (transaction log is on this array)
sdd = 6x150GB 10krpm in RAID 10 (database is on the array)
we're using ReiserFS on each of the arrays as the file system
OS is debian Etch x64
raid controller = 3ware 9650 12port - 256MB cache
8GB RAM, core 2 duo - quad core 2.4ghz 8MB L2 Cache
it would seem like the io subsystem is the limiting factor, but i feel like we should be barely hitting a wall, you can see from the example its writing < 2MB/s to the array. If i try to copy a big file (1-2GB) over to the array, the array hits 300MB/s sustained, it can burst write files < 256mb AT 800MB/S
Here's some of our settings
shared_buffers = 1024MB
temp_buffers = 32MB
max_prepared_transactions = 50
work_mem = 64MB
maintenance_work_mem = 256MB
max_stack_depth = 7MB
max_fsm_pages = 512000
vacuum_cost_delay = 20ms
fsync = on
wal_buffers = 1MB
checkpoint_segments = 32
effective_cache_size = 4128MB
If you guys have any suggestions it would be greatly appreciated
-Evan Reiser
Re: High IOWAIT times, low iops? Can't figure out what the bottleneck is...
From
"Evan Reiser"
Date:
Here is the table structure for the table with all the inserts
id(PK) integer
dateinserted datetime without timezone
dateexamined datetime without timezone
lockedby integer
done boolean
failed boolean
invalid boolean
>select * from table limit 1;
id | done | failed | dateinserted | dateexamined | lockedby | invalid
--------+------+--------+---------------------+----------------+----------+---------
3130902 | f | f | 1900-01-01 00:00:00 | | 0 | f
There is a clustered index on dateinserted,
+ a non clustered index on id
+ a non clustered index on done,failed,invalid
+ a non clustered index on lockedby
There are no triggers
id(PK) integer
dateinserted datetime without timezone
dateexamined datetime without timezone
lockedby integer
done boolean
failed boolean
invalid boolean
>select * from table limit 1;
id | done | failed | dateinserted | dateexamined | lockedby | invalid
--------+------+--------+---------------------+----------------+----------+---------
3130902 | f | f | 1900-01-01 00:00:00 | | 0 | f
There is a clustered index on dateinserted,
+ a non clustered index on id
+ a non clustered index on done,failed,invalid
+ a non clustered index on lockedby
There are no triggers
On 7/3/07, Evan Reiser <evan.reiser@gmail.com> wrote:
I was wondering if you guys have some suggested settings for our server, i think we are not hardware limited but the configureation is set up incorrectly. For some reason our database seems to have trouble handling 5-10+ inserts per second which seems to be a pretty trivial load for this hardware, we're seeing very high %iowait, this is a pretty typical output for #iostat -m 5
avg-cpu: %user %nice %system %iowait %steal %idle
0.97 0.00 0.82 89.69 0.00 8.52
Device: tps MB_read/s MB_wrtn/s MB_read MB_wrtn
sda 85.40 0.18 0.72 0 3
sdb 0.00 0.00 0.00 0 0
sdc 30.43 0.00 0.76 0 3
sdd 93.91 0.71 0.12 3 0
sda = 2x320GB 7200rpm in RAID1 (operating system)
sdc = 2x150GB 10krpm in RAID1 (transaction log is on this array)
sdd = 6x150GB 10krpm in RAID 10 (database is on the array)
we're using ReiserFS on each of the arrays as the file system
OS is debian Etch x64
raid controller = 3ware 9650 12port - 256MB cache
8GB RAM, core 2 duo - quad core 2.4ghz 8MB L2 Cache
it would seem like the io subsystem is the limiting factor, but i feel like we should be barely hitting a wall, you can see from the example its writing < 2MB/s to the array. If i try to copy a big file (1-2GB) over to the array, the array hits 300MB/s sustained, it can burst write files < 256mb AT 800MB/S
Here's some of our settings
shared_buffers = 1024MB
temp_buffers = 32MB
max_prepared_transactions = 50
work_mem = 64MB
maintenance_work_mem = 256MB
max_stack_depth = 7MB
max_fsm_pages = 512000
vacuum_cost_delay = 20ms
fsync = on
wal_buffers = 1MB
checkpoint_segments = 32
effective_cache_size = 4128MB
If you guys have any suggestions it would be greatly appreciated
-Evan Reiser