Brian Knutsson in StorageTips and TricksvSphere

Storage Optimization for VMware vSphere

This is meant as a dynamic article for looking up best practice settings for different storage arrays when adding them to VMware.

Why modify the default settings?

When datastores are added to an ESXi host, there are multiple ways that ESXi can leverage the storage. In some cases ESXi will use Most Recently Used path (Active/Standby or MRU) by default, which means that you only leverage one path at the time. This could result in a bottleneck in your storage infrastructure. Many arrays are able to handle Round Robin (Multi path Active/Active or RR) By enabling this will distribute your storage traffic onto multiple adapters, provided that you have multiple adapters.

Other settings can involve how many I/O ESXi should send to a path before switching to another path, or advanced settings that alters the way ESXi handles the storage.

Getting these settings correct will most often result in better performance, but can also help you stay out of trouble that can lead to breakdowns.

How can you determine what settings to modify?

Most often storage vendors like HP will publish documentation or whitepapers about the settings they recommend and support.

I will add the different storage arrays that I come across including where in the documentation the settings can be found. It is wise to check the documentation once in a while, since the best practices sometimes change. I do not give guaranties for the settings I post here, but I will try and reference vendor documentation, so that you will be able to check for yourself.

I will try to use claim rules and global settings where ever possible, since this will also include storage added at a later time, like if you decide to create a new LUN/Datastore from the same array. But this also means that if you connect a new type of array, you will have to consider the settings that you set previously.

What is a claim rule?

A claim rule is like a pattern that ESXi will look at when adding disk devices, and if the pattern matcheds, it will use the settings defined in the rule.

Storage Array List

This will eventually be a list of different storage arrays, and the settings needed to optimize ESXi behavior around this array.

HP 3PAR Arrays

References:

HPE 3PAR VMware ESX/ESXi Implementation Guide

Controlling LUN queue depth throttling in VMware ESX/ESXi

Relevant ESXi versions: 5.1u1+, 5.5, 6.x

Claimrules:

This claim rule will look for a vendor string “-V” called “3PARdata” and then enable tpgs, and set the iops limit to 1 I/O for each path. That means that ESXi will send one I/O down a path before switching to a new path. It will also change the array from MRU to RR.

# CLI
esxcli storage nmp satp rule add -s "VMW_SATP_ALUA" -P "VMW_PSP_RR" -O iops=1 -c "tpgs_on" -V "3PARdata" -M "VV" -e "HP 3PAR Custom iSCSI/PC/FCoE ALUA Rule"

# Powershell
$vmhost = Get-VMHost <HostName>
$esxcli = Get-EsxCli -VMHost $VMHost.Name
$esxcli.storage.nmp.satp.rule.add($null,"tpgs_on","HP 3PAR Custom iSCSI/PC/FCoE ALUA Rule", $null, $null, $null, "VV", $null, "VMW_PSP_RR","iops=1", "VMW_SATP_ALUA", $null, $null, "3PARdata")

The claim rules only apply to luns added in the future. To change existing luns you have to change them to Round Robin policy.

This can be accomplished using PowerCLI command.

Set all 3PAR Luns to Round Robin policy:

$VMHost = Get-VMHost <hostname> 
$VMHost | Get-ScsiLun -LunType disk | Where {$_.CanonicalName -like "naa.6000*"} | Set-Scsilun -MultiPathPolicy RoundRobin

Set Round Robin Luns IOPS Limit for host:

$VMHost = Get-VMHost <hostname> 
$VMHost | Get-ScsiLun -LunType disk | Where-Object {$_.Multipathpolicy -like "RoundRobin"} | Set-ScsiLun -CommandsToSwitchPath 1 | Select-Object CanonicalName, MultipathPolicy, CommandsToSwitchPath

Check Round Robin Luns IOPS Limit for host:

$VMHost = Get-VMHost <hostname> 
$VMHost | Get-ScsiLun -LunType disk | Where-Object {$_.Multipathpolicy -like "RoundRobin"} | Select-Object CanonicalName, MultipathPolicy, CommandsToSwitchPath

More information can be found here: https://kb.vmware.com/kb/2069356

You can also change the default policy:

# Set default PSP profile and default SATS name using Powershell
$vmhost = Get-VMHost <HostName>
$esxcli = Get-EsxCli -VMHost $VMHost.Name
$esxcli.storage.nmp.satp.set($null,"VMW_PSP_RR","VMW_SATP_ALUA")

ESXi Advanced Settings:

Disk.QFullThreshold = 4
Disk.QFullSampleSize = 32

# Set using Powershell
$VMHost = Get-VMHost <HOST NAME>
$VMhost | Get-AdvancedSetting Disk.QFullThreshold | Select -First 1 | Set-AdvancedSetting 4 -Confirm:$false
$VMhost | Get-AdvancedSetting Disk.QFullSampleSize | Select -First 1 | Set-AdvancedSetting 32 -Confirm:$false

Alternatively settings it on each disk/device:

Use this method if you have different storage arrays attached.

esxcli storage core device set --device device_name --queue-full-threshold 4 --queue-full-sample-size 32

IBM V7000

As far as i can tell you do not need to do anything except make sure your datastores are configured to use the round-robin algorithm.

Ref: https://www.redbooks.ibm.com/redbooks/pdfs/sg247521.pdf

DELL EMC Unity Storage Arrays

References:

Dell EMC Unity Storage with VMware vSphere Look at pages 9 and 14

Relevant ESXi versions: 6.x

Claimrules:

This claim rule will look for a vendor string “-V” called “DGC” and then enable tpgs, and set the iops limit to 1 I/O for each path. That means that ESXi will send one I/O down a path before switching to a new path. It will also change the array from MRU to RR.

esxcli storage nmp satp rule add -s "VMW_SATP_ALUA_CX" -V "DGC" -P "VMW_PSP_RR" -O "iops=1" --claim-option="tpgs_on"

#PowerCLI Commands
$VMHost = Get-VMHost <hostname>
$esxcli = $VMHost | Get-ESXCli
$esxcli.storage.nmp.satp.rule.add($null,"tpgs_on","Dell EMC Unity", $null, $null, $null, $null, $null, "VMW_PSP_RR","iops=1", "VMW_SATP_ALUA_CX", $null, $null, "DGC")

The claim rules only apply to luns added in the future. To change existing luns you have to change them to Round Robin policy.

This can be accomplished using PowerCLI command.

Set all Unity Luns to Round Robin policy:

$VMHost = Get-VMHost <hostname> 
# I do not yet know what the CanonicalName pattern is, so you will have to check for your self.
$VMHost | Get-ScsiLun -LunType disk | Where {$_.CanonicalName -like "naa.?000*"} | Set-Scsilun -MultiPathPolicy RoundRobin

Set Round Robin Luns IOPS Limit for host:

$VMHost = Get-VMHost <hostname> 
$VMHost | Get-ScsiLun -LunType disk | Where-Object {$_.Multipathpolicy -like "RoundRobin"} | Set-ScsiLun -CommandsToSwitchPath 1 | Select-Object CanonicalName, MultipathPolicy, CommandsToSwitchPath

Check Round Robin Luns IOPS Limit for host:

$VMHost = Get-VMHost <hostname> 
$VMHost | Get-ScsiLun -LunType disk | Where-Object {$_.Multipathpolicy -like "RoundRobin"} | Select-Object CanonicalName, MultipathPolicy, CommandsToSwitchPath

More information can be found here: https://kb.vmware.com/kb/2069356

Jumbo Frames

If iSCSI is utilized jumbo frames should be enabled according to documentation. MTU should be set to 9000.

Delayed Ack

To avoid unwanted delays when the storage is under heavy load, Delayed Ack should be disabled. I cannot confirm whether this is still an issue on vSphere 6.7, but I think. Better safe than sorry.

Guide to disabling Delayed Ack can be found here: https://kb.vmware.com/s/article/1002598