Wednesday, October 31, 2012

Linux VM IO scheduler tuning

We had a Linux VM (CentOS 5.4 2.6.18-194.17.1.el5) with intermittent performance issues running on ESXi 5.0.  The VM runs java build jobs and looked to be IO bound during some of these builds.

We opened a VMware case on the issue to determine what could be done improve the performance.
One of the engineer's recommendations is to disable RedHat's IO scheduler as it can interfere with the ESXi scheduler.
This is the explanation cited by the engineer:
http://www.fatmin.com/2010/03/redhat-io-scheduler-configuration-in-a-virtual-machine-.html


So currently  our test VM dev-app-01's disks are all cfq

[fletch@dev-app-01 ~]$ cat /sys/block/sda/queue/scheduler
noop anticipatory deadline [cfq]
[fletch@dev-app-01 ~]$ cat /sys/block/sdb/queue/scheduler
noop anticipatory deadline [cfq]
[fletch@dev-app-01 ~]$ cat /sys/block/sdc/queue/scheduler
noop anticipatory deadline [cfq]
[fletch@dev-app-01 ~]$ cat /sys/block/sdd/queue/scheduler
noop anticipatory deadline [cfq]

I made the noop update and check out the before and after

BEFORE:

[root@dev-app-01 ~]# dd if=/dev/sdc of=/dev/null bs=1M count=5000
5000+0 records in
5000+0 records out
5242880000 bytes (5.2 GB) copied, 60.597 seconds, 86.5 MB/s



AFTER (disable Linux GOS IO scheduler):

[root@dev-app-01 ~]# echo "noop" > /sys/block/sda/queue/scheduler
[root@dev-app-01 ~]# echo "noop" > /sys/block/sdb/queue/scheduler
[root@dev-app-01 ~]# echo "noop" > /sys/block/sdc/queue/scheduler
[root@dev-app-01 ~]# echo "noop" > /sys/block/sdd/queue/scheduler
[root@dev-app-01 ~]# dd if=/dev/sdc of=/dev/null bs=1M count=5000
5000+0 records in
5000+0 records out
5242880000 bytes (5.2 GB) copied, 1.73969 seconds, 3.0 GB/s
[root@dev-app-01 ~]# dd if=/dev/sdc of=/dev/null bs=1M count=5000
5000+0 records in
5000+0 records out
5242880000 bytes (5.2 GB) copied, 1.44314 seconds, 3.6 GB/s
[root@dev-app-01 ~]# dd if=/dev/sdc of=/dev/null bs=1M count=5000
5000+0 records in
5000+0 records out
5242880000 bytes (5.2 GB) copied, 1.50344 seconds, 3.5 GB/s

Has anyone experienced similar IO performance before and after disabling the OS scheduler?


No comments: