vCSA upgrade from 5.0u1b to 5.1a Part II

First time running VMware's vCenter Server Appliance upgrade, it failed miserably because my vCSA was a rsyslog location (initial failure here).

So I waited a bit to try again, and hey look, there's an "a" update to vCenter 5.1 for upgrade bugs!

Here's my basic steps so that I didn't fail again in the upgrade:

• Stop rsyslog from hosts
• rm -rf on the contents of /opt/vmware/var/log/remote 
• Obviously you could copy them to another system if you needed to keep them around, lucky for me this is a dev environment.
• TAKE SNAPSHOT NOW
• Export your permission set unless you just let anyone get into your vCSA..
• Grab new appliance ova, install and set for another IP in the same subnet as the 5.0 appliance 
• Start up and go through the key exchange under the "upgrade" choice on the 5.1 appliance
• Because of my previous dumping of tasks/events/history the upgrade took just a few minutes
• The source 5.0 appliance is powered down and the 5.1 appliance takes over the IP of the original (the root password was duplicated as well, that was nice). I think it rebooted as well. 
• I needed to reauth to the AD domain (removed old computer account, add this one)
• My AD permissions were not copied over in the transfer, blank slate with only root access to anything. 
• I needed to poke Active Directory LDAP settings  to be able to get SSO working and then I could import the custom permission sets and AD groups back into the appliance. 

Whew. Better than last time? Maybe. The fact that the vCSA is *still* a linux VM without NTP running but connected to AD is a big miff for me. I'll save that for the other post I'm dreaming up..

root ssh on vCSA (whoops!)

I'm going to make a post with some hardening and vmware-linux-best-practices-that-vmware-has-not-done of VMware's vCenter Server Appliance (5.1), but only after I stop locking myself out...

If you turn off root ssh before creating another real user in the appliance, do not be alarmed. You can head into the admin web interface and change the setting on the Admin tab (Toggle SSH setting). The Administrator SSH logon enabled will change from no to yes (assuming you haven't locked yourself out of the web admin as well). :)

Oh, and the new skitch sucks.

vCSA upgrade from 5.0u1b to 5.1

First try on upgrading to the new 5.1 appliance and a big fail with a filled destination 5.1 vCSA root partition. (( Upgrade that worked here ))

If you have configured remote syslog on your source appliance, this routes to a /new directory on the destination appliance during the upgrade (not /storage/log). Unlucky for us, VMware choose not to have this /new directory with it's own mount point.

My work around is to create a new 20G drive and mount to /new before starting.

Oh, and take a snapshot BEFORE importing/swapping keys between source and destination vCSAs..

I'll update if I come across any other gotchas.


I've pinged VMware's kb with this, hopefully they'll get something up quickly. As for me...to last night's backup I go. Good thing this isn't production. 

vCSA DB2 shenanigans before upgrade

My initial post on upgrading the vCenter appliance to 5.0u1b build 804277 gave some good tips that I found around and figured out myself.
But I'll tell you, dear reader, that the upgrade process kept failing for me. Too much db2 export which I narrowed down to tasks & events.

I had set tasks and events to be kept for 180 days (this might be the default). I bumped this down to 30 days hoping that the rows would be deleted and then my upgrade would work again. This didn't seem to happen. So this post is about clearing your embedded db2 database of history stats, tasks and events before upgrade. Note that it doesn't look like anything in the db2 cli is case sensitive.

I should mention that I am *not* a DBA, that this could hose your installation, it will most definitely hose your stats, events and tasks so this process might not be a good idea in your production environment. The vCSA is a VM: Use snapshots to your advantage here.

• SSH into the appliance as root. Turn off vcenter services:

# service vmware-vpxd stop

• Change your login to db2inst1, get into the db2 cli, connect to the vCenter database:

# su - db2inst1
~> db2
db2 => connect to VCDB

• I couldn't remember if I had bumped the transaction logsize on this appliance, so I did the command anyway: 

db2 => update db configuration FOR VCDB USING logprimary 16 logsecond 112 logfilsiz 8192

• Delete contents of the history tables:

TRUNCATE TABLE vc.vpx_hist_stat1 IMMEDIATE
TRUNCATE TABLE vc.vpx_hist_stat2 IMMEDIATE
TRUNCATE TABLE vc.vpx_hist_stat3 IMMEDIATE
TRUNCATE TABLE vc.vpx_hist_stat4 IMMEDIATE
TRUNCATE TABLE vc.vpx_sample_time1 IMMEDIATE
TRUNCATE TABLE vc.vpx_sample_time2 IMMEDIATE
TRUNCATE TABLE vc.vpx_sample_time3 IMMEDIATE
TRUNCATE TABLE vc.vpx_sample_time4 IMMEDIATE

•  Deleting the contents of the tasks & events is not as straightforward. There are foreign keys associated with these tables that need to be removed before you can delete the rows and then re-create the foreign keys. 
• See how many rows of data: 

select count(*) from vc.vpx_event
select count(*) from vc.vpx_event_arg
select count(*) from vc.vpx_task

• Remove foreign keys: 

alter table vc.VPX_EVENT_ARG drop constraint FK_VPX_EVENT_ARG_REF_EVENT
alter table vc.VPX_EVENT_ARG drop constraint FK_VPX_EVENT_ARG_REF_ENTITY
alter table vc.VPX_ENTITY_LAST_EVENT drop constraint FK_VPX_LAST_EVENT_EVENT
alter table vc.VPX_EVENT drop constraint FK_VPX_CHANGE_TAG
alter table vc.VPX_EVENT drop constraint FK_VPX_EVENT_REF_COMPUTERES
alter table vc.VPX_TASK drop constraint FK_PARENT_TASK_REF
alter table vc.VPX_TASK drop constraint FK_VPX_TASK_CHANGE_TAG
alter table vc.VPX_TASK drop constraint FK_VPX_TASK_REF_ENTITY

• Delete data: 

truncate table vc.VPX_TASK immediate
truncate table vc.VPX_ENTITY_LAST_EVENT immediate
truncate table vc.VPX_EVENT immediate
truncate table vc.VPX_EVENT_ARG immediate

• Re-add foreign keys: 

alter table vc.VPX_EVENT_ARG add constraint FK_VPX_EVENT_ARG_REF_EVENT foreign key(EVENT_ID) references vc.VPX_EVENT (EVENT_ID) on delete cascade
alter table vc.VPX_EVENT_ARG add constraint FK_VPX_EVENT_ARG_REF_ENTITY foreign key (OBJ_TYPE) references vc.VPX_OBJECT_TYPE (ID)
alter table vc.VPX_ENTITY_LAST_EVENT add constraint FK_VPX_LAST_EVENT_EVENT foreign key(LAST_EVENT_ID) references vc.VPX_EVENT (EVENT_ID) on delete cascade
alter table vc.VPX_EVENT add constraint FK_VPX_CHANGE_TAG foreign key(CHANGE_TAG_ID) references vc.VPX_CHANGE_TAG(CHANGE_TAG_ID) on delete set null
alter table vc.VPX_EVENT add constraint FK_VPX_EVENT_REF_COMPUTERES foreign key(COMPUTERESOURCE_TYPE) references vc.VPX_OBJECT_TYPE(ID)
alter table vc.VPX_TASK add constraint FK_PARENT_TASK_REF foreign key(PARENT_TASK_ID) references vc.VPX_TASK(TASK_ID)
alter table vc.VPX_TASK add constraint FK_VPX_TASK_CHANGE_TAG foreign key(CHANGE_TAG_ID) references vc.VPX_CHANGE_TAG(CHANGE_TAG_ID) on delete set null
alter table vc.VPX_TASK add constraint FK_VPX_TASK_REF_ENTITY foreign key(ENTITY_TYPE) references vc.VPX_OBJECT_TYPE(ID)

•  Your tables are now empty:

select count(*) from vc.vpx_event
select count(*) from vc.vpx_event_arg
select count(*) from vc.vpx_task

•  quit from db2 cli, exit from db2inst1 user, restart vcenter services to see if everything powers back up: 

=> quit
~> exit
# service vmware-vpxd start

•  Log into the vSphere Client to check that everything looks good. 
• Try your upgrade again. For me, just truncating the history stats still failed after exporting the data for ~18 hours. Yeah. 18 hours and then I had to revert to snap. Dumping the events/tasks was the only way this was ever going to work. 

Thanks to vfrank, juanma and IBM's db2 documentation.

vCSA upgrade tips & recommendations

Upgrading the vCenter Server Appliance looks simple enough. If you run into problems, maybe these tips can help:

• Create the 20G (yes 20G) disk in the kb article. 
• Take a snapshot first (shut down services)
• Really. TAKE A SNAPSHOT FIRST
• If the upgrade does not go smoothly the first time, do not try again with a half-broken install. Revert to snap.
• If your upgrade breaks during the export, check your free inodes (df -i)
• Increase the number of inodes on the /storage/db/export partition (mkfs -t ext2 -i 8192 /dev/sdc1 )
• Problems mounting /dev/sdc1 (doesn't like ext2)? tune2fs -O has_journal /dev/sdc1
• tail -f /opt/vmware/var/log/vami/updatecli.log during the upgrade. This file is created a minute or two after you kick off the upgrade. 
• There will be a long (potentially hours long) pause in the log while the database export is happening. Top will show db2bp kjourneld and db2sync in use. Do not kill any of these processes. 
• You didn't take my advice and create a snapshot? You needed to use a restored backup? You might need to remove the /etc/udev/rules.d/70-persistent-net.rules mac addresses if the appliance says it can't bring up eth0. 

vDR email spam

Crazy crazy amounts of email spewing from a vDR 2.0 appliance. VMware says this is a known bug and you could turn off email alerts and/or reboot your appliance on a regular basis. Drink Up.

Come on  guys, really? That's not acceptable. How about I limit the number of outgoing packets per day on port 25? It worked this morning... but long term I hope that VMware is fixing this bug.

iptables -A OUTPUT -p tcp --dport 25 -m limit --limit 10/day -j ACCEPT

Also I was trying to find a config file that would have the settings for the email reports, but perhaps this isn't kept in a config or xml file. find / -mtime -1 after I made changes in the UI didn't seem to point to any text files.

ipcop 2.0.2

IPCOP2 still needs buslogic SCSI controller; and in order to use BusLogic, make sure you've chosen Other Linux 32bit.

And E1000 nics (but you should have known that).

old linux VM trick..

I was re-deploying a vDR2.0 machine today and could *not* get the console keyboard to stop stuttering. I looked up the old trick for delay, but it doesn't seem to be taking effect. The options->general->config params are not available for this appliance, not sure if the machine is ignoring other options on the vmx file.. I ended up using the Microsoft on-screen keyboard to get in and set the IP. Here's the text of the kb article for future personal reference. I haven't had to use this hack in years.

http://kb.vmware.com/kb/196

To reduce these effects, increase the time threshold necessary for auto-repeat in the remote console.

Power off the virtual machine. Add a line, similar to this, at the end of your virtual machine's configuration (.vmx) file:

keyboard.typematicMinDelay = "2000000"

The delay is specified in micro-seconds, so the line in the example above increases the repeat time to 2 seconds. This should ensure that you never get auto-repeat unless you intend it.

Power on the virtual machine. To make the changes using the vSphere Client: Power off the virtual machine Right click virtual machine select Edit Settings Click Options > General > Configuration Parameters Click Add Row Under Name enter keyboard.typematicMinDelay In the Value field 2000000 Click OK Power on the virtual machine

vSphere 5 vCenter Appliance first take

UTC. Do not change the timezone on the appliance or the embedded DB2 database will not start. 

Also watch for the DB2 logsize: http://kb.vmware.com/kb/1005259

And the Likewise AD auth needs to be a domain admin, but it won't put the hostname in correctly if it's longer than 15 chars. 

Bugs to work out? Nahhhhh..

Other than those...few..gaping holes, I think VMware's got something going for it.