This here is a story about a well executed HCI cluster upgrade that went slightly off-script towards the tail end of it. What we were dealing with was a hyperconverged, 4-node, vSphere cluster running about 80-90 VM’s. Now there are some complexities to consider with the HCI part, but in the end we are indeed performing a hypervisor upgrade to ESXI nodes which results in a VMware Tools upgrade for all the VM’s hosted on the cluster nodes.
Fast forwarding to the Tools upgrades, all seemingly routine and upgraded effortlessly. However, there’s always one. Why is there always one? Initially, things looked like a failed attempt to mount the installer but after poking at it, it became apparent that there was a little more going on.
After some troubleshooting, what was observed was a Tools instance that could not be upgraded, reinstalled, repaired, or removed using any of the preferred tactics. Now what? The VM was running and VM Tools seemed functional but clearly not in an ideal state. I suppose others might leave it, but that’s “dirty” and it would most definitely catch up with you later. A decision was made to do the right thing which often comes with some level of additional pain and suffering.
Don’t get ahead of yourself though, have a backup, a clone, or something else in your back pocket to recover from, before you unleash the beast. Only then should you get your BFH and start swinging.
Thankfully, some Googling linked me to some other poor soul who battled something similar and so with some confidence and Plan B readily available, I proceeded forward and resolved by going down the path of manual extraction of VMware tools.
What’s described below is a combination of steps from a forum post, some intuition, and maybe some luck. You’re mileage may vary, so have a backup!
- Rebooted the VM
- Stopped VMware Tools services in services.msc
- Dropped to to a PS session: sc.exe delete vmtools
- Deleted the following folders and all content:
- C:\Program Files\VMware
- C:\Program Files\Common Files\VMware
- C:\Program Data\VMware
- C:\users\%username%\App data\VMware
- Used regedit.exe [HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\services] and deleted all VMware related drivers including:
- vmci
- vm3dmp
- vmaudio
- vmhgfs
- vmmemctl
- vmmouse
- VMRawDisk
- VMTools
- vmusbmouse
- vmvss
- vmware physical disk helper services
- VMwareCAF*
- Deleted VMware.Inc using regedit.exe [HKEY_LOCAL_MACHINE\SOFTWARE\VMware.Inc]
- Deleted related drivers under C:\Windows\System32\Drivers\
- Removed the VMware SVGA driver from Device Manager
- Rebooted the VM
- Manually installed VMware Tools
