Storage Optimization for VMware vSphere

This is meant as a dynamic article for looking up best practice settings for different storage arrays when adding them to VMware.

Why modify the default settings?

When datastores are added to an ESXi host, there are multiple ways that ESXi can leverage the storage. In some cases ESXi will use Most Recently Used path (Active/Standby or MRU) by default, which means that you only leverage one path at the time. This could result in a bottleneck in your storage infrastructure. Many arrays are able to handle Round Robin (Multi path Active/Active or RR) By enabling this will distribute your storage traffic onto multiple adapters, provided that you have multiple adapters.

Other settings can involve how many I/O ESXi should send to a path before switching to another path, or advanced settings that alters the way ESXi handles the storage.

Getting these settings correct will most often result in better performance, but can also help you stay out of trouble that can lead to breakdowns.

How can you determine what settings to modify?

Most often storage vendors like HP will publish documentation or whitepapers about the settings they recommend and support.

I will add the different storage arrays that I come across including where in the documentation the settings can be found. It is wise to check the documentation once in a while, since the best practices sometimes change. I do not give guaranties for the settings I post here, but I will try and reference vendor documentation, so that you will be able to check for yourself.

I will try to use claim rules and global settings where ever possible, since this will also include storage added at a later time, like if you decide to create a new LUN/Datastore from the same array. But this also means that if you connect a new type of array, you will have to consider the settings that you set previously.

What is a claim rule?

A claim rule is like a pattern that ESXi will look at when adding disk devices, and if the pattern matcheds, it will use the settings defined in the rule.

Storage Array List

This will eventually be a list of different storage arrays, and the settings needed to optimize ESXi behavior around this array.

HP 3PAR Arrays

References:

HPE 3PAR StoreServ Storage and VMware vSphere 6 best practices Look at pages 10 and 35

Controlling LUN queue depth throttling in VMware ESX/ESXi

Relevant ESXi versions: 5.1u1+, 5.5, 6.x

Claimrules:

This claim rule will look for a vendor string “-V” called 3PARdata” and then enable tpgs, and set the iops limit to 1 I/O for each path. That means that ESXi will send on I/O down a path before switching to a new path. It will also change the array from MRU to RR.

esxcli storage nmp satp rule add -s "VMW_SATP_ALUA" -P "VMW_PSP_RR" -O iops=1 -c "tpgs_on" -V "3PARdata" -M "VV" -e "HP 3PAR Custom iSCSI/PC/FCoE ALUA Rule"

The claim rules only apply to luns added in the future. To change existing luns you have to change them to Round Robin policy.

This can be accomplished using PowerCLI command.

Set all 3PAR Luns to Round Robin policy:

$VMHost = Get-VMHost <hostname> 
$VMHost | Get-ScsiLun -LunType disk | Where {$_.CanonicalName -like "naa.6000*"} | Set-Scsilun -MultiPathPolicy RoundRobin

Set Round Robin Luns IOPS Limit for host:

$VMHost = Get-VMHost <hostname> 
$VMHost | Get-ScsiLun -LunType disk | Where-Object {$_.Multipathpolicy -like "RoundRobin"} | Set-ScsiLun -CommandsToSwitchPath 1 | Select-Object CanonicalName, MultipathPolicy, CommandsToSwitchPath

Check Round Robin Luns IOPS Limit for host:

$VMHost = Get-VMHost <hostname> 
$VMHost | Get-ScsiLun -LunType disk | Where-Object {$_.Multipathpolicy -like "RoundRobin"} | Select-Object CanonicalName, MultipathPolicy, CommandsToSwitchPath

More information can be found here: https://kb.vmware.com/kb/2069356

ESXi Advanced Settings:

Disk.QFullThreshold = 4

Disk.QFullSampleSize = 32

Alternatively settings it on each disk/device:

Use this method if you have different storage arrays attached.

esxcli storage core device set --device device_name --queue-full-threshold 4 --queue-full-sample-size 32

Finishing words

You can create your own claim rules. Vendor and Model names can be found by using the following command:

esxcli storage core device list

Notice: There is an issue with VMware ATS feature on different storage devices. According to the VMware KB vSphere 6.5 is not having this issue.

HP StoreVirtual

HP 3PAR

IBM SVC and Storwise

VMware KB

VMware Storage Best Practices

I hope you found this article helpful, if you find a settings that is outdated, or wrong do not hesitate to write a comment below.

Leave a Reply

Your email address will not be published. Required fields are marked *