Recently on a a few of our more I/O intensive linux machines we noticed dramatic iowait increases, especially during the backup window. While they are quite heavily used boxes, and the backup will always hit the disks hard, it was more than I would have expected!
While the server itself was still perfectly responsive, throughput did drop to around 7/8mb during these times, and the googling began!
Firstly we looked at our Equalogic SAN’s, and ran some iometer tests with the same average workload on a physical machine with one volume on a SAN and on a VM. The volume connected directly to the SAN performed as expected, while the VM started to struggle slightly, not miles off what you would expect, but the difference was there! Using esxtop we were able to check the latency on the requests, and how big the disk queue was, all looked normal with around 2/3ms latency. While we checked a few other bits and bobs, I came across a few recent posts on the paravirtualised drivers available for ESX, which have only recently become properly supported.
Over the last week we have started using the PVSCSI driver available in vSphere, which has significantly improved performance above the old LSI parallel adapter. While not yet supported for boot disks on RHEL, it is on windows 2003 and 2008, and I would definately recommend trying this adapter out for any machines with a more signifcant I/O demand.
There is a great tutorial for RHEL at http://southbrain.com/south/tutorials/installing-redhat-enterprise-5.html and I will be adding a windows howto shortly!