do not use lockdown mode in vmware

PSA: Do not use Lockdown Mode on your ESXi servers.  Yesterday I ran into a major headache.  One of my ESXi 5 hosts went down and it happened to the be hosting my Virtual Center VM.  My Host Isolation Response is set to Shutdown which should have gracefully powered off my Virtual Center server and started it up another host.  Except that it didn’t; it stayed powered off.  Being the security minded SysAdmin that I am, I had Lockdown Mode enabled on all my hosts.  Huge Mistake.

If Lockdown Mode is enabled you literally cannot manage the host at all unless it’s through the vSphere Client connected to your Virtual Center server.  Literally.  Like not even through the console via KVM, DRAC, iLO, etc.  So you can have physical access to the ESXi server but you absolutely cannot disable Lockdown Mode or interact with the host in anyway that would allow you to get your server back up.  Unbelievable.

After calling into support and confirming this their solution was to install a new ESXi server, connect it to my storage, and cold migrate the vCenter server to it.  This is the only time VMware has let me down.  This all could have been fixed if they allowed you to disable Lockdown Mode from the console.  If I have KVM or physical access to the host anyway, it’s game over security-wise VMware.  Fix this!

EDIT:  I should state that I still have a ticket open with VMware to figure out why my vCenter Server wasn’t started back up on another host in my HA cluster.  Also, if your Virtual Center isn’t virtualized it’s less of a headache, but still very stupid that you cannot disable Lockdown Mode from the console.

users getting locked out after domain upgrade

This is fairly obscure, but who knows it may help someone down the road.  Last Friday we started a domain upgrade from 2003 to 2008 R2.  We got two of our three DC’s upgraded and saved the last one for Monday morning.  When we came in Monday all hell broke lose.  Users could log in, but when they opened Outlook they were prompted for credentials and even if they entered them correctly Outlook would just keep prompting them.  Same thing if they tried to browse file shares on our DFS.  The Help Desk was getting hammered and after a couple hours and a call to Microsoft we found the problem: 

HKEY_LOCAL_MACHINE\System\CurrentControlSet\Control\Lsa\LmCompatibilityLevel

Long story short, this key on all our XP clients (yes we are still on XP…) is set to 0 and on the new DC’s it was set to 5.  Whenever a user logged in they were immediately locked out hence the prompting of creds.  We changed the key on the DC’s to 2 one at a time rebooting each afterwards and then everything was fine.  Phew.

More info on the LM levels…

Read More

cluster service on passive node stuck on “starting”

If you have a passive cluster node who’s Cluster Service is stuck on ‘starting’ and you can’t remote desktop the active node (i.e does nothing after entering credentials to log) you basically just need to reboot the active node.  You can first try restarting the Cluster Service on the active node using Computer Management on the passive node and connecting to the active node.  A hard reboot should do the trick failing that.  In my experience, the passive node’s Cluster Service will come right up and seize all the cluster resources as soon as the active node goes down.

One thing I haven’t tried, and didn’t think about until writing this article, is to disable the Heartbeat NIC and possibly the normal LAN NIC on the active node and see what happens.  This could possibly work as well but I’m not sure.