vCSA 5.1u1b saves the day

Windows shop install of a "simple install" vCenter. Piece of cake and the local IT staff should be able to manage it in the long term. Except they don't put the new vCenter server into DNS and I don't realize it. >_< "Error 29155.Identity source discovery error" but install of SSO continues so I think it's something we can work with. Turns out, not so much but it was a pair of bad decisions by me that turned into a 5 hour headache and wanting to do bodily harm to the manager who decided that SSO should be required for all vCenter deployments. Hindsight #1: It's a VM. Use snapshots before mucking with things or installing. "Nah, it'll be fine, simple install...right?" Hindsight #2 and a bit: Do not uninstall SSO. Do not try to re-install everything from the "simple vCenter install" and expect it to work. Clean system or nothing. vCSA 5.1u1b (which the local IT staff won't know what to do with) was downloaded, installed, configured in about an hour. Setting the appliance for AD auth automatically set the AD domain in SSO (although with insecure settings) but it worked and I didn't have to muck with it. Saved the day. So thank you VMware for investing time into the vCSA and screw you for bad SSO error messages (or just SSO in general) and uninstall procedures that don't clean the system correctly. You cost me an entire Saturday afternoon and into the evening. Should I manually check DNS for the new vC server that I was handed? Yes. Should the SSO installer have continued upon error? NO.

New Cluster Name

While I've seen a whole host of boring cluster names in my travels, I like to open it up to voting. The ESXi cluster name is just a fairly arbitrary text value in the database that can be changed anytime so why not have some fun with it. :)

Please vote once, you can choose as many names as you like. Obvious ballot stuffing will disqualify the entry.

Dell R805 SD install of ESXi 5.1

I have a stack of oldish Dell PowerEdge R805 servers that have been running ESX and then ESXi 5.0 on a standard hard drive configuration. Now that they'll not be in production use I was giving them a once over and saw they had an internal 1G SD card that was off in the BIOS.

Hmmm.

I decided to turn it on and install ESXi 5.1 on the SD card. I'm pretty sure that the version of ESXi preinstalled was the OEM embedded Dell 3.0 version of ESXi. The 5.1 installer throws quite the fit when doing the check of what's running in the card but it looks like it installs and runs just fine. Mostly python and libparted, see screenshot.

"The selected storage device contains ESX(i). The version could not be determined. Only installation is possible." Fine by me.

root cause analysis log hunting

I was asked to unravel the all-to-common filled datastore and the response from the local IT staff that ultimately made the company pull a critical server from backup (the actions of the local IT staff that is).

My problem? "Task: Delete file" from the datastore browser.
What file?

Looks like the verbose and trivia[l] vpxa.log would have this information ...but... the logs I have don't go quite far enough back. vpxa is CHATTY. Search the log for nfc.

Now, I was almost lucky in this case because the datastore was only used by two hosts. In a larger environment, which ESXi log would have this info? Hopefully whichever host the guest was associated with, but again.. unraveling what host a guest was on at a particular point could be time consuming.

Hopefully I have better luck next time and the client asks for this analysis closer to when it happened!

5.1u1 rant. Too much fail in readme

5.1U1 might not work if you're attached to lots of AD groups, so use Windows passthrough 
Oh, sorry. I'll do that now... Oh wait.
I can't login with Windows passthough because of SSO config issues. NetBIOS? REALLY?
Or maybe my login issue is that SSO seems to be only hanging onto to UUID and not the username so when my account was deleted I get locked out.
Or I use the vCSA and it doesn't allow Windows passthrough (<--see comment for KB fix).
Or I (some would say correctly) setup a SSO load balanced HA configuration, and it doesn't allow windows passthrough either. ARG.

I've been reading the upgrade readmes from VMware for years now and none of them seem to have as much fail as this one. No Windows Auth anymore for SQL Server? What? 
That's ok I guess, since I couldn't install the schema in the first place.

So many of the issues, work arounds or otherwise identified issues have to do with special characters and non-English strings. Why is there no standard lib for those crazy double quotes in passwords? Really!?!

Who the hell is coding this? I've been to VMware HQ...which building basement houses those monkeys?

Enjoy. It would make for a good drinking game whenever SSO, special characters or database issues are mentioned. Please, not all three at once...you'll get alcohol poisoning.

Can't even install vSphere DP? This is not a good sign

Downloaded current 5.1.1 (build 5.1.56.179) of vDP 0.5T ova. 

From C# Windows Client, able to get through the ovf wizard, fails on creation disk 1 of 4:
Unable to access file:
<path>\vSphere_Data_Protection_-_0.5TB-disk1.vmdk.
Unable to connect to the remote server

On the web client, after RDP'ing to a Windows host that had access on port 9443 and installed Flash and installed the client integration plugin (arg), the wizard wouldn't let me pass the Select Storage. I said Next it went back to Select Storage. I hit Setup Network, it went back to Select Storage.

I think the issue here is that the 0.5T appliance is actually more like 900G when thick and the largest datastore I have in this environment is 700G (and yes, it's empty for this purpose).

I am choosing thin provision during the wizard but obviously this isn't good enough for vDP.

From the horror stories in the community forums I'm just going to say screw it.

This is a big issue when I move my production vCenter to 5.1. I'm only backing up 5 VMs with vDR in prod (why? because it's unreliable) but at least I can install (and re-install and re-install) it.

vCSA update 5.1.0a to 5.1.0b

Just a few hiccups when upgrading to 5.1.0b... Nothing like the koolaid I had to mix up (here and here) to get from 5.0U1a to 5.0U1b to 5.1a.
I read the release notes and rebooted the VM at the end of the update.

Then nothing worked.
WTF?

I logged into the console of the VM and saw that it had reverted to the temporary IP that I gave the appliance when I initially made the upgrade from 5.0 to 5.1. This may be due to the vApp options -> Properties still having the temp IP... I just changed it to the correct IP in /etc/sysconfig/network/ifcfg-eth0 (then ifdown eth0 / ifup eth0).

This worked for the main services, but not, it seems for SSO. I can't login to the web client, the error message clearly shows that it's trying to talk on the temp IP. I expect another reboot (or actually a shutdown/startup so that I can change the vApp settings) needs to happen for vmware-sso to start listening on the right address (restarting the vmware-sso did nothing).

/etc/vmware-sso/ls_url.txt shows the correct IP (lookup service)
/etc/vmware-vpx/vpxd.cfg did not. I restarted vmware-vpxd and still couldn't login.. a restart of the vmware-client service did the trick (and while it's called vmware-client, it did not quit active sessions with the C# client).

So I'm not sure if my editing of the vpxd file and the restart of vmware-client was needed or just the restart of vmware-client... at least this post will point someone in the right direction if they have the same issue.

The other fun thing was the ESXi hosts disconnected to vCenter and needed to be manually connected back. Not sure if this was because of the IP address change or something else.