vCSA update 5.1.0a to 5.1.0b

Just a few hiccups when upgrading to 5.1.0b... Nothing like the koolaid I had to mix up (here and here) to get from 5.0U1a to 5.0U1b to 5.1a.
I read the release notes and rebooted the VM at the end of the update.

Then nothing worked.
WTF?

I logged into the console of the VM and saw that it had reverted to the temporary IP that I gave the appliance when I initially made the upgrade from 5.0 to 5.1. This may be due to the vApp options -> Properties still having the temp IP... I just changed it to the correct IP in /etc/sysconfig/network/ifcfg-eth0 (then ifdown eth0 / ifup eth0).

This worked for the main services, but not, it seems for SSO. I can't login to the web client, the error message clearly shows that it's trying to talk on the temp IP. I expect another reboot (or actually a shutdown/startup so that I can change the vApp settings) needs to happen for vmware-sso to start listening on the right address (restarting the vmware-sso did nothing).

/etc/vmware-sso/ls_url.txt shows the correct IP (lookup service)
/etc/vmware-vpx/vpxd.cfg did not. I restarted vmware-vpxd and still couldn't login.. a restart of the vmware-client service did the trick (and while it's called vmware-client, it did not quit active sessions with the C# client).

So I'm not sure if my editing of the vpxd file and the restart of vmware-client was needed or just the restart of vmware-client... at least this post will point someone in the right direction if they have the same issue.

The other fun thing was the ESXi hosts disconnected to vCenter and needed to be manually connected back. Not sure if this was because of the IP address change or something else.

ESXi 3.5 build 207095 to 5.0 current build steps

A client has the free version of ESXi 3.5 using local storage and wanted to move up to a current version of ESXi. I wasn't able to find a one-stop shop that said how to get from A -> E and for a while there I thought I could at least skip 4.0...nope. If you're not using local storage, please do yourself a favor and just do a fresh install.

It's possible that there's a slightly more efficient way to do this, but this was the way I was guaranteed not to loose the local datastore data. vmware.com was giving me some grief when I was trying to finish this post so I don't have links up to some of the packages. You should be able to search around and find them.

Things you'll need:
- vMA (current version is fine)
- vSphere Client 4.0 with Host Update Utility (here VMware-viclient-all-4.0.0-208111.exe)
- vSphere 4.0 update 1 upgrade package (here ESXi-4.0.0-1.9.208167-upgrade-release.zip)
- vSphere 4.0 to 4.1 upgrade package (here upgrade-from-esxi4.0-to-4.1-update03-800380.zip)
- vSphere 5.0 iso (standard one here VMware-VMvisor-Installer-5.0.0.update02-914586.x86_64.iso)
- vSphere 5.0 current build (here) if needed

Things you might want:
- vSphere 4.0 upgrade guide (here)
- vSphere 4.1 upgrade guide (here)

Ready?
- SCP the 4.1 patch to your vMA
- SCP the 5.0 patch to the ESXi datastore * 
- Login to vMA
- Make a backup copy of your host config via vMA

vicfg-cfgbackup --server <ESXi-host-ip> --portnumber <port_number> --protocol
<protocol_type> --username username --password <password> -s <backup-filename>

- Check that there's a scratch partition setup on your host via vMA

vicfg-advcfg --server <ESXi-host-ip> --username root -g ScratchConfig.ConfiguredScratchLocation
if not found:
vifs --server <ESXi-host-ip> --username root --mkdir "[datastore1] .locker-esxtest"
vicfg-advcfg --server <> --username root -s /vmfs/volumes/datastore1/.locker-esxtest ScratchConfig.ConfiguredScratchLocation

- Shutdown VMs and put into maintenance mode

vicfg-hostops --server <ESXi-host-ip> --operation enter

- Start up the Host Update Utility, add your host and patch with the 4.0u1 upgrade package.
-- Yeah. 4.0u1. Not current version of 4.0, that gave back an error that it wasn't compatible.
- Wait a bit. I'm not sure what files are created when, but there was a directory called /vmupgrade that was created and then deleted while the HUU stayed at 3%
- One or two reboots. My test ESXi system rebooted at 3% and 34%, prod ESXi only did the reboot at 34%.
- ESXi reboots to 4.0, HUU eventually will say that the upgrade failed when installing packages after the initial base reboot. Do not fret...keep moving.
- You can login via the vSphere Client & test that everything still looks good.
- Log into vMA and run vihostupdate with the 4.1 upgrade against the host.

 vihostupdate --server <ESXi-host-ip> -i -b <zip bundle>

- Reboot again.
- Test again after starting up into 4.1.
- Reboot system and boot to the 5.0 CD.
- The installer will see your current install and happily upgrade.
- And another reboot. I hope your server doesn't have too much memory that it's set to check upon each reboot..
- Back to vMA and run the esxcli command to patch

esxcli software profile update -d <depot url or offline bundle zip file> -p <profile_name>

- Reboot
- Yes, you are done. Luckily, ESXi patches are cumulative so once you're at a major revision you only need to grab the most recent patch.


* If, like me, you've drawn a blank on how to get ssh enabled on ESXi 3.5:
From console: 
- Alt-F1 unsupported
- enter root pw
- vi /etc/inetd.conf -> unhash #ssh
- ps | grep inetd
- kill -HUP <processID>

vCSA SSO Secret Active Directory Handshake

---
The test connection button was working for me with "reuse session" but I was getting "LDAP: error code 8 - 00002028: LdapErr: DSID-0C0901FC, comment: The server requires binds to turn on integrity checking if SSL\TLS are not already active on the connection" and "Failed to serialize response" when I actually said to use that configuration.

Error messages were in /var/log/vmware/sso/ssoAdminServer.log 

---
vCSA and working for me:

Primary server URL:
ldaps://FQDN:636
Base DN for users:
<Only the domain, no other DN/OUs> DC=ad,DC=company,DC=com
Domain name:
ad.company.com
Base DN for groups:
<same as users>

cert (*.cer) created for me from the FQDN AD/GC server listed in Primary server URL.
(VMware KB pointing to MS KB)

Auth type:
Password
Created a "service account" user to connect.


The client needs the full ad.company.com\user, not shorthand domain to get in.

vCSA upgrade from 5.0u1b to 5.1a Part II

First time running VMware's vCenter Server Appliance upgrade, it failed miserably because my vCSA was a rsyslog location (initial failure here).

So I waited a bit to try again, and hey look, there's an "a" update to vCenter 5.1 for upgrade bugs!

Here's my basic steps so that I didn't fail again in the upgrade:

• Stop rsyslog from hosts
• rm -rf on the contents of /opt/vmware/var/log/remote 
• Obviously you could copy them to another system if you needed to keep them around, lucky for me this is a dev environment.
• TAKE SNAPSHOT NOW
• Export your permission set unless you just let anyone get into your vCSA..
• Grab new appliance ova, install and set for another IP in the same subnet as the 5.0 appliance 
• Start up and go through the key exchange under the "upgrade" choice on the 5.1 appliance
• Because of my previous dumping of tasks/events/history the upgrade took just a few minutes
• The source 5.0 appliance is powered down and the 5.1 appliance takes over the IP of the original (the root password was duplicated as well, that was nice). I think it rebooted as well. 
• I needed to reauth to the AD domain (removed old computer account, add this one)
• My AD permissions were not copied over in the transfer, blank slate with only root access to anything. 
• I needed to poke Active Directory LDAP settings  to be able to get SSO working and then I could import the custom permission sets and AD groups back into the appliance. 

Whew. Better than last time? Maybe. The fact that the vCSA is *still* a linux VM without NTP running but connected to AD is a big miff for me. I'll save that for the other post I'm dreaming up..

root ssh on vCSA (whoops!)

I'm going to make a post with some hardening and vmware-linux-best-practices-that-vmware-has-not-done of VMware's vCenter Server Appliance (5.1), but only after I stop locking myself out...

If you turn off root ssh before creating another real user in the appliance, do not be alarmed. You can head into the admin web interface and change the setting on the Admin tab (Toggle SSH setting). The Administrator SSH logon enabled will change from no to yes (assuming you haven't locked yourself out of the web admin as well). :)

Oh, and the new skitch sucks.

Export / Import vCenter permissions.

These scripts, written by Alan and LucD, have been invaluable to me. I now have them set to run weekly as backups for production and I can do a manual export for my dev environment if needed (and that's where I've used the import, twice now, with vCSA upgrade issues).
Thank you, Darren for posting them up, and yes! they still work for vSphere 5.1!

I did need to change the domain from the shorthand to longhand version in the xml file importing into 5.1 from 5.0, but this might have been the way I put the domain in SSO.

Drink Up!

vCSA 5.1 SSO LDAP issues

Working with the vSphere vCenter Server Appliance 5.1 (build 880472, the "a" edition) I could not get straight LDAP (not LDAPS) to work for Single Sign On. Was not happening. Not with Anonymous, not with username, not with reuse session. Anonymous was just broken, but it looks like this is by design with current AD and is not recommended (by anyone, anywhere, but if you search for it you can find out how to do it).
Error message is [LDAP: error code 8 - 00002028: LdapErr: DSID-0C0901FC, comment: The server requires binds to turn on integrity checking if SSL\TSL are not already active on the connection, data 0, v1db1].
VMware says "Hey crazy! Get your SSL certs in order!" (Drink the kb here).
I said "That's great VMware, what are my other options?" Turns out I answered my own question year(s) ago!  (Old Koolaid here)

That's right, just turn off the cert requirement:
Computer configuration - Policies - Windows Settings - Security Settings - Local Policies/Security Options - Domain Controller: LDAP server signing requirements (None/Require signing/Undefined[which is the same as None])  -- Change to None

Not as secure, but I could import my users/groups into the new appliance (luckily had a backup) without making other changes (create cert template, export, blah blah) that my local domain admins didn't want to do on a Friday afternoon. 

See updated secret sso cert handshake post for secure settings that worked for me

Pic to show straight (insecure) LDAP settings.
ldap://<fqdn of an active directory server>:389
CN=Users,DC=ad,DC=<part2>,DC=<part3>
full domain name
reuse session

The default "Users" OU is listed as a CN. If you point to another root OU in your domain, you would use OU=<blah> instead of CN. You can also screw the CN/OU and pull everything from the domain as well by putting in just the DC parts.