Today I upgraded a customer to ESXi 6.5 Update 1, but unfortunately some of them ended up purple screening at reboot after they were updated.
Affected Servers so far
- HPE BL460c Gen9
- HPE DL360p Gen8 (Reported by anonymous user)
- HPE DL380 Gen9 (Reported by Bernhard)
- HPE DL380 Gen8 (Reported by Ralf)
- HPE DL380p Gen9 (Reported by Victor)
PSOD Error
PSOD: #PF Exception 14 in world 68297:sfcb-intelcim IP 0x41801b704d8f addr 0x443919649c000
In short the customer was running ESXi 5.5, and I reinstalled the servers using HPE 6.5 Update 1 OEM Image found here: https://my.vmware.com/web/vmware/details?downloadGroup=OEM-ESXI65U1-HPE&productId=614
I configured the servers and found that VMware Update Manager had 3 additional updates:
Oddly it still wanted to install Update 1. I had already updated some Gen8 servers, and they had no issue with me installing all updates, so I went ahead and installed all three updates. But in this case the server never came back online. Instead it kept giving a PSOD on boot:
After some investigation I found it to be caused by the Ixgben driver updates. It is a driver for Intel network adapters. This also corresponded with the PSOD coming from sfcb-intelcim.
Anyway I contacted HPE Support, and the supported just heard of a similar case, so he wanted some logs and promised to return to me asap.
HPE Support return a little later and told me that they do not have a record of the noted kernel exception, so I was probably among the first to experience the error, if not the first. Hooray I was the first!
Workaround
HPE Returned and stated that those two updates should just be excluded, since it is a driver delivered by intel, and it does not work with the HPE firmware.
You should also avoid the VMware 6.5 Update 1 in Update Manager. I guess it also contains a ixgben driver from intel.
To quickly return to a working installation without reinstalling. you can press Shift+R during startup to get back to a working configuration. Thank you to my college André Briand de Crévecoeur for bringing this to my attention. https://kb.vmware.com/kb/1033604
Same Problem here, exactly the same 1:1 HP DL-380 G9
Maybe I was the second? This article saved me a lot of troubleshooting, thanks!
I am happy to help, and congrats on your silver medal. 🙂
Same issue – DL360p G8’s
exactly the same problem – DL380p G9
Any resolution to this issue? We are experiencing the same problem.
Yes, as the article explains, just exclude the 3 updates in VUM.
Same problem here on HPE DL380p Gen8
I was able to workaround this issue by removing the Ixgben meta data and vib files inside the installer package.
It still shows it needs updated in the vSphere Update Manager, but using vmware -lv shows the correct information now.
+1
DL380G8, installed with VMware-ESXi-6.5.0-Update1-5969303-HPE-650.U1.10.1.0.14-Jul2017.iso.
Updated with VUM, which installed U1 again and two ixgben packages. -> PSOD
Excluding the packages in VUM is not my favorite solution, but I guess there is still no other way.
If we exclude the HPE updates and just applying the vSphere 6.5 update 1 Rollup from 27/07 I get the same error.
This into a ProLiant BL460c Gen8 blades.
Hi we have the same issue DL380 G9. I would like to point out that the NICs we use are 561FLR-T Adapters. Oddly enough we have another cluster with 561T Adapters that does not show these symptoms..
We had the same problem on 2 DL380 Gen9 with 10GB NIC’s present.
Thank you for sharing this workaround.
Does everyone, who has the problem, have 10GB NIC’s in their systems? Or is it also happening when there are no 10GB NIC’s present in the systems?
Its happening with or without 10gb nic. Its related to the intel adapters. If you dont have intel (HP servers have integrated broadcom adapters) you dont run into this issue.
Same problem and a thousand others, I’ll never and never buy another *deleted word* hp server, in my career I have never experienced such a lot of headaches with servers as with hp. Long life to DELL and other brands.
I have the same issue. After using the HP 6.5 custom ISO & then updating the server PSOD. I Contacted VMware support and was told after a day of the tech investigating that it might be the net-ixgbe drivers. I tried that and still PSOD. I removed everything but the Update 1 and PSOD. So I removed the two drivers & the Update 1 and it worked. When VMware support gets Back to me I will let them know and see if I can get this fast tracked.
Who remembers esXpress backup solution??? 😉
Removing the Intel-cim provider VIB worked for me, too. After removing the VIB and rebooting, I was able to install all the patches through VUM.
I am curious, though, what effect will removing that VIB? What functionality will be lost (if any), by not having the intel-cim installed?
Ridiculous that this is still an outstanding issue 1/11/2018. I re-installed an HP twice not knowing about the SHIFT+R trick. I wonder if Dell/EMC just doesn’t mind leaving this broken since HP is a competitor. It’s funny right Dell/EMC?
I agree with MG, I’ve been waiting to update some servers until this was fixed. Guess I have no choice other than to work around the problem now.
Still seeing the same problem on 1/29/2018
I just upgraded some HPE BL460c Gen9 servers today, and I did not have this problem. Installed update 1 and ixgben driver without problems.
Thank you Ed Donnelly,
You solution worked perfectly fine for HP DL380s Gen8
Using HP Provided 6.5 Custom ESXi image build, then running updates from VUM