ESXi: Slow Disk Performance on HPE Gen8

I had an old, but reliable HP Microserver N40L. I had been using it 24/7 since 2012. It was running ESXi 5.1. There was a couple of 2003R2 servers (a DNS and a WEB ones). A sort of encapsulation of dangerous vectors. It was just a separate information web server. If hacked, it was easy to recover from a backup and to restart. Normally it was not connected to the internal network but for getting content updates. But time passed and my server got “tired”.

I wanted to run both DNS and WEB at least on Windows Server 2008R2 and ideally on Windows Server 2012 R2. N40L started hanging up. N40L didn’t have enough hardware resources. You couldn’t install something more powerful than a 1.5 GHz AMD Turion (™) II processor with 2/2 architecture and 8 GB of RAM to it.

I understand that HP Microserver Gen8 is not a magic pill as well, but you can install 16 GB of memory and, for example, a 2.3/3.5 GHz Intel® Xeon® E3-1220l processor with 2/4 architecture and ultra-low energy consumption (about 17-20 W). You can install, for example, Intel® Xeon® E3-1270 3.4/4.1 GHz with 4/8 architecture. But then you shouldn’t be surprised that the processor alone is 1.5 times more expensive than the Microserver :-).

HP Microserver Gen8 is the last HP microserver, which:

• can be upgraded;
• has an iLO card.

Unfortunately, Gen9 did not appear, and the new Gen10 is … just read about it and you’ll see 🙂

HP Microserver Gen8 is not sold officially, but you still can purchase it. I have bought a used Gen8 (G2020T). Then I have installed the HPE ESXi 5.5U3 custom image on it.

However, it turned out there was more about it. The most interesting things started to happen afterwards.

Installed: Vmware-ESXi-5.5.0-Update3-3568722-HPE-550.9.6.5.9-Dec2016.iso

Disk driver version: scsi-hpvsa-5.5.0.100-1OEM.550.0.0.1331820

As it turned out, HP has spoiled something in the disk subsystem driver for ESXi 5.5 and disk management became not as efficient as it should. Moreover, as I found out later, this problem also occurred in the HPE ESXi 6.0, 6.5 and 6.7 images.

After talking to my mates and searching the Web, I understood that the matter was in the driver that HPE integrated in its custom image with the installer of ESXi 5.5 or later.

However, this problem can be solved. The Internet community (https://homeservershow.com) has managed to find a driver that really increases the disk performance in HP Microserver Gen8.

Driver version: scsi-hpvsa-5.5.0-88OEM.550.0.0.1331820

You can download the driver for free from the official HPE website:

Now we need to install it. The procedure is described below. First of all, check the version of the installed driver, and if it is different, replace it for the right one.

Connect ESXi host console using PuTTY, authenticate as root and run this command:

esxcli software vib list | grep scsi

This is what I had prior to changing the driver:

esxcli software vib list | grep scsi

scsi-hpsa 5.5.0.124-1OEM.550.0.0.1331820 HPE VMwareCertified 2018-04-10
scsi-hpdsa 5.5.0.52-1OEM.550.0.0.1331820 Hewlett-Packard PartnerSupported 2018-04-10
scsi-hpvsa 5.5.0.100-1OEM.550.0.0.1331820 Hewlett-Packard PartnerSupported 2018-04-10
scsi-mpt2sas 15.10.06.00.1vmw-1OEM.550.0.0.1198610 LSI VMwareCertified 2018-04-10
scsi-bfa 3.2.6.0-1OEM.550.0.0.1331820 QLogic VMwareCertified 2018-04-10
scsi-bnx2fc 1.713.20.v55.4-1OEM.550.0.0.1331820 QLogic VMwareCertified 2018-04-10
scsi-bnx2i 2.713.10.v55.3-1OEM.550.0.0.1331820 QLogic VMwareCertified 2018-04-10
scsi-qla4xxx 644.55.37.0-1OEM.550.0.0.1331820 QLogic VMwareCertified 2018-04-10

It means not the right one. Why? Here is what the disk performance test showed. Not exactly a test, but you can understand what is being tested from the given commands.

Run the following commands in ESXI console:

cd /vmfs/volumes/[datastore]
time dd if=/dev/zero of=tempfile bs=8k count=1000000

Here is the result:

1000000+0 records in
1000000+0 records out
real 14m 12.62s
user 0m 12.23s
sys 0m 0.00s

Not so bad, is it?

Compare it with the results obtained for the same configuration with ESXi 5.1U3 installed:

1000000+0 records in
1000000+0 records out
real 17m 25.62s
user 0m 7.23s
sys 0m 0.00s

As you can see, there is an improvement compared to the previous ESXi version. However, you’ll have to trust me and then look at the different result. Read this post to the end.

So, let’s change the driver.

The procedure is quite simple. It is supposed that you have downloaded the driver you need from HPE website following the link above.

1. Stop all running VMs;
2. If disabled, enable SSH;
3. Copy scsi-hpvsa-5.5.0-88OEM.550.0.0.1331820.x86_64.vib to /tmp (e. g., using WinSCP);
4. Connect to ESXi host console using PuTTY;
5. Change the current folder to the one you put the file to, i. e. to /tmp:
   cd /tmp
6. Copy the VIB file to the folder, from which it will be installed:
   cp scsi-hpvsa-5.5.0-88OEM.550.0.0.1331820.x86_64.vib /var/log/vmware/
7. Enable Maintenance Mode of the host:
   esxcli system maintenanceMode set --enable true
8. Remove the current driver of the disk subsystem:
   esxcli software vib remove -n scsi-hpvsa -f
9. Install the right scsi-hpvsa-5.5.0-88OEM driver from the file: esxcli software vib install -v file:scsi-hpvsa-5.5.0-88OEM.550.0.0.1331820.x86_64.vib --force --no-sig-check --maintenance-mode
10. Restart ESXi, disable Maintenance Mode, disable SSH (if necessary) and start your virtual machines.

Is it easy? Yes, it is.

But you always need to make sure if the author was not lying. Let’s make sure that the driver version has changed:

esxcli software vib list | grep scsi

scsi-hpsa 5.5.0.124-1OEM.550.0.0.1331820 HPE VMwareCertified 2018-04-10
scsi-hpdsa 5.5.0.52-1OEM.550.0.0.1331820 Hewlett-Packard PartnerSupported 2018-04-10
scsi-hpvsa 5.5.0-88OEM.550.0.0.1331820 Hewlett-Packard PartnerSupported 2018-04-10
scsi-mpt2sas 15.10.06.00.1vmw-1OEM.550.0.0.1198610 LSI VMwareCertified 2018-04-10
scsi-bfa 3.2.6.0-1OEM.550.0.0.1331820 QLogic VMwareCertified 2018-04-10
scsi-bnx2fc 1.713.20.v55.4-1OEM.550.0.0.1331820 QLogic VMwareCertified 2018-04-10
scsi-bnx2i 2.713.10.v55.3-1OEM.550.0.0.1331820 QLogic VMwareCertified 2018-04-10
scsi-qla4xxx 644.55.37.0-1OEM.550.0.0.1331820 QLogic VMwareCertified 2018-04-10

Yes, it has changed to the right one. Then I started the performance test again. I was taken aback by the result:

cd /vmfs/volumes/[datastore]
time dd if=/dev/zero of=tempfile bs=8k count=1000000

1000000+0 records in
1000000+0 records out
real 2m 6.73s
user 0m 5.21s
sys 0m 0.00s

It is SEVEN times faster than with the previous driver and almost 9 times faster than in ESXI 5.1U3.

Forum users confirmed that the wrong driver is installed during the installation of ESXi 6.0 and 6.5. If you replace it with scsi-hpvsa-5.5.0-88OEM.550.0.0.1331820, the disk subsystem began to work as fast as in the last of my tests.

In my opinion, this is a very convincing argument in favor of replacing the ESXi storage driver.