vKoolAid

The VCP5 can smell fear

2012-02-22T15:33:00.001-08:00

For me, it was the fact that I'd never actually gone through an AutoDeploy setup. I mean, Enterprise Plus license level only with large shops that are deploying massive hosts? Not used often. I use the ESX Deployment Appliance. The every-person's version of kickstart for ESXi systems. I love EDA.

Anyway. The test could sense that I had skimmed the AutoDeploy documentation. I'm sure it has sensed fear from others on different subjects. A respectable 400 considering that I probably missed all the autodeploy questions.

hostdatastoresystem.createvmfsdatastore for object datastoresystem error

2012-02-13T18:50:00.001-08:00

Well, from the ramblings of other blogs and the community forums, that ^ error string from the vSphere Client could mean just about anything could be wrong with your storage, your LUNs or your storage network.
In my case, after a long call with Dell EqualLogic and VMware, it turned out to be MTU size. In order to make my world right again, the MTU size on the Dell 6224 switches needed to be higher than the MTU size on the surrounding infrastructure. So: vSwitch, VMKernel and EqualLogic mtu: 9000. 6224's: 9126

Previous to that, there were logins and logouts on the iSCSI side, all sorts of errors seen in the vmkernel logs, but nothing...absolutely nothing...pointed in the direction of the MTU.

So, my short story that was actually really long is to show you, dear reader, that the first blog post that you find with this error message might not be your problem, it might not be the third or fifth solution you find. You might need to get help from someone else to check your configuration for something stupid that you've missed because you are less familiar with certain switches.

ipcop 2.0.2

2012-02-01T10:46:00.000-08:00

IPCOP2 still needs buslogic SCSI controller; and in order to use BusLogic, make sure you've chosen Other Linux 32bit.

And E1000 nics (but you should have known that).

Legacy Applications

2012-02-01T09:50:00.000-08:00

I got it. I've got a better handle on "cloud" than ever before. Aaron Delp and Brian Gracely helped me there, putting it in perspective of "traditional" hypervisor/cloud stacks. Drink up: Drink Me. Google their names and great things pop up. From here in the trenches, it's good to get a high level overview every once and a while.

old linux VM trick..

2012-01-24T15:45:00.000-08:00

I was re-deploying a vDR2.0 machine today and could *not* get the console keyboard to stop stuttering. I looked up the old trick for delay, but it doesn't seem to be taking effect. The options->general->config params are not available for this appliance, not sure if the machine is ignoring other options on the vmx file.. I ended up using the Microsoft on-screen keyboard to get in and set the IP. Here's the text of the kb article for future personal reference. I haven't had to use this hack in years.

http://kb.vmware.com/kb/196

To reduce these effects, increase the time threshold necessary for auto-repeat in the remote console.

Power off the virtual machine.Add a line, similar to this, at the end of your virtual machine's configuration (.vmx) file:

keyboard.typematicMinDelay = "2000000"

The delay is specified in micro-seconds, so the line in the example above increases the repeat time to 2 seconds. This should ensure that you never get auto-repeat unless you intend it.

Power on the virtual machine.To make the changes using the vSphere Client:Power off the virtual machineRight click virtual machine select Edit SettingsClick Options > General > Configuration ParametersClick Add RowUnder Name enter keyboard.typematicMinDelay In the Value field 2000000Click OKPower on the virtual machine

you do not hold privilege "system > read" on virtual machine ""

2011-12-09T13:00:00.001-08:00

Yep, blank name for the virtual machine.

More information:
Call storageresourcemanager.configurestoragedrsforpod blah blah
(didn't write it all down)

I did once get the above ^ permission error and then this message:
The guest os id is not valid. Therefore, editing these vm settings is not recommended.

What was I doing?
Cloned a virtual machine and then tried to make any changes to the settings (I tried changing the port group and adding/removing a vCD iso). Saying OK to commit the changes brings up that permission error.

I looked up the Storage Resource Manager parts in the API docs and there are no associated privileges: Drink MeSimilar calls need System.View priv set (which this user holds).
Here's the way I've currently been able to get rid of the error: I removed the associated datastores from the datastore clusters (moved it to the parent folder) and now I can't recreate the problem. I moved the datastores back under the datastore clusters (without otherwise making changes) and still no error message.

Sounds like a bug with storage DRS to me... I'll wait to see if it shows back up.

Consolidated server and desktop workload licensing failure

2011-10-31T13:48:00.000-07:00

VMware View and a standard vSphere server workload deployment cannot mix. Separate physical hosts, separate vCenters. WHY???
If you have a SMB who wants to get their feet wet with both virtualized Desktop and Server workloads, you're telling them that they need to purchase twice the rackspace to have the same n+1 cluster setup. I can see large shops wanting to keep the workloads (and admins in charge of them) as separate entities, but don't make it a restriction.

From VMware View 5 Licensing FAQ:
Q: Can I run other server workloads on the vSphere that is included in View?
A: No. The vSphere and vCenter components of the View bundle are restricted to desktop deployments. A desktop virtual machine is defined as a virtual machine running the following operating systems: Windows 95/98, Windows 2000 Professional, Windows XP Professional, Windows Vista Ultimate, Windows Vista Business, or Windows Vista Enterprise, Windows 7.
The only exception to this rule are the components that make up the virtualized desktop infrastructure (View Manager, vCenter Server, another Connection Broker and/or any desktop management and performance monitoring tools used solely for hosted desktop virtual machine(s) within an operating system of a server).

vSphere 5 vCenter Appliance first take

2011-10-18T17:15:00.000-07:00

UTC. Do not change the timezone on the appliance or the embedded DB2 database will not start.

Also watch for the DB2 logsize: http://kb.vmware.com/kb/1005259

And the Likewise AD auth needs to be a domain admin, but it won't put the hostname in correctly if it's longer than 15 chars.

Bugs to work out? Nahhhhh..

Other than those...few..gaping holes, I think VMware's got something going for it.

ESX 3.0 on ESX 4.0

2011-10-11T18:25:00.001-07:00

I was trying to run ESX 3.0.3 U1 inside of my vSphere 4.0 implementation (so that I could test upgrade strategies to 4.1).

I set the OS as RHEL 4 (64bit) and specified "Use Intel VT-x/AMD-V" under the CPU/MMU Virtualization Options area (rather than automatic). It seemed to boot fine at that point.
Note that the install went fine without poking in the settings area, but it was just hanging at "vmkernel has been loaded successfully" rather than actually booting into the OS.

Now I can get to work.

VMW 2011 sessions

2011-09-30T17:14:00.000-07:00

Using the bash script linked below will download all available flv presentations or the session pdf if no flv was available. Some sessions did not have any content (about a dozen or so). Resulting folder size is currently 16.5G. Requires lwp-download that should work on sol/nix/OSX ..as long as perl is installed.
Or just browse the script for sessions you want. Session name will be the file name.

...if anyone asks, i got it from a friend of a friend.

Drink Me

virtualappsonline redirect

2011-08-01T15:58:00.000-07:00

I'm going to redirect a site I own and have done nothing with over here.. It's virtualappsonline.com

The only real topic on that site was a VM I made for the marketplace in Jan 2007. This is the text, just for fun:

activeCollab 0.7.1 Project Management Solution
Description
From http://www.activecollab.com website:
activeCollab is an easy to use, web based, open source collaboration and project management tool. Set up an environment where you, your team and your clients can collaborate on active projects using a set of simple, functional tools. 100% free!
Download links at the bottom of this post
Highlights
1. Easy to install and easy to use
2. Web-based. After installation only thing you'll need to use it is web browser
3. 100% open source, 100% free
4. No limitations on number of projects, clients, team members...
5. Per project permissions
---
Google for other reviews of this project management platform. Most reviews compare it to Basecamp by 37signals.
---
Below is text from included readme file:
==Debian Settings==
Login to server: root toor
Login to mysql: root toor
ac 4ct1v3
Login to activeCollab website: admin 4ct1v3
* mysql database ac created for activeCollab
* ssh access available to root
* no firewall settings
* vmware tools not installed (no GUI installed)
==Virtual Machine Settings==
floppy1, sound adapter turned off
ethernet1 bridged (will receive a DHCP address from your local network). To change network settings to static, see the /etc/network/interfaces file.
==Getting the Server / activeCollab Setup==
change /var/www/activecollab/config/config.php ROOT_URL listing to the server's FQDN for active collab to work
/etc/apt/sources.list should be changed to a mirror closer to you. See http://www.debian.org/mirror/list for a listing.
Dotdeb should stay in the sources list for php5 / mysql5 updates. http://www.dotdeb.org/
Operating system: Debian Sarge 3.1 Stable
Applications installed:
Debian Sarge 3.1: Apache2-mpm-prefork, MySQL-server, PHP5, PHP5-MySQL, PHP5-gd, activeCollab 0.7.1
VMware Tools installed: No - With no GUI, VM tools doesn't change performance.
Size: 222.3 MB MB
Primary account
Username: root
Password: toor
Memory allocated: 256MB
==Download Links==
Available upon request... but come on now..
md5sum: ad8904b13d56f5c6c6221fb8f9ce6e60

The KoolAid is now laced

2011-07-14T10:26:00.000-07:00

The potassium cyanide has been added. Who will drink the new vSphere 5 licensing model?

For myself, it's time to look into XenServer and maybe even HyperV. vSphere 4.1 is an excellent product and hopefully VMware will reconsider this idiotic move.

--- Update 8/3/11:

Yep. They fumbled the ball but have come up with a solution that should work for most shops. Smart move VMware. XenServer is pretty easy to run.

Drink Me now :)

vCenter database growth problem and fix

2011-07-01T18:18:00.000-07:00

Symptom: Disk space low after a scheduled job at night in dev environment. DBA & I were alerted, DBA looked into it initially.

Short Term Cause: Short term cause (figured out by DBA) was when the index for the VPX_HIST_STAT2 table is re-organized (when fragmentation > 20%). The table was huge (42,000,000+ rows, while prod holds 7,000,000+) and holds the weekly history stats. This table, when running correctly, will hold the last week's worth of stats based on the stat level associated..and nothing else. I changed the stat level in dev to reflect the same settings as prod, restarted services and did some general poking with no success in a reduction of the table size.

Long term cause: When looking at the "VPXV_HIST_STAT_WEEKLY" view in dev, there were dates listed all the way back to 2009, although nothing very synchronous. Further digging in SQL Server brought me to the sql agent job that is supposed to do the rollup sequence and purge old records from that table ("Past Week stats rollup"). At some point (unknown date), the third step that purges the old record was set to never run (the second step would quit the job upon success). I shut down vCenter services and ran the third step in the job to start purging the records.

Hiccup: The purge job grew the transaction log to the point that it filled up the partition, grinding SQL Server to a halt. I was trying to cancel the job/shut down the sql server agent service, when the gears stopped. I added a new 25G disk to the VM and moved the log file (then at 11G) to this new partition. I started SQL Server back up and banged on it a bit to start the virtualCenterDev database up again*. Once up, I re-started the rollup/purge job and had to run to an appointment.

Finale: Watched by DBA, the job completed successfully then ran again successfully & quickly on the next schedule (probably because the vcenter service was still off). DBA ran a full backup then shrank the data & log files on both the vcenter db & tempdb. Ran another full backup & t-log backup. The log file is still on the new partition and can stay there through the lifespan of this VM (not long, as 4.1 is imminent).
vCenter services are now on and the database is at it's [probably correct] size of 1.2G.

Moral of the Story: Perhaps I should pay better attention to the database size. There is an estimator built into the vCenter client on the statistic level settings page for the db size based on the stat levels you choose. vcenterdev is now at that estimated size (rather than the 4.2G with one table running at 2.2G).

* : database offline, sp_detach_db, sp_attach_DB... http://www.mssqltips.com/tip.asp?tip=1774

vCenter update manager version 4.0.0.7313

2011-05-09T12:48:00.000-07:00

Version 7313 is 4.0 U3.
I've got two vCenter instances running in the same subnet... sometimes I'm not sure if I'm connecting to the right discovered plug-in. It's only a problem when I go to upgrade the dev/staging environment. Heh.

Please use vCenter. Please?

2011-04-07T19:02:00.000-07:00

I've learned that I've been living the good life with vCenter (formerly known as virtualcenter). It's always been a part of the environments under my care. Now I'm seeing that other shops are not as lucky... I keep having to look up ways to hack around the things that I've taken for granted all this time. Like alarms, historical graphs, cloning... I could go on. Luckily the hacks are out there, it's just not pretty.
You can get 3 vSphere Essentials lics + vcenter for under $1k. Now, that cheap version of vcenter has some gotchas like database size, but worth it. So worth it. So all you smallish shops: Use vCenter, please. Don't run it as a VM *on* the hosts it's managing. You'll thank me later.

vCenter management over WAN links

2011-03-22T15:23:00.000-07:00

Traffic that heads through vCenter in a WAN vSphere architecture:

- vUM patches
- Template Clones (that live in one location that you are deploying in another)
- virtual console
- linking virtual CDs

Some tips with syncing options listed here: Drink Me

While PHD's free patch downloader can help with lack of local vUM.

Other than connecting locally to the ESX(i) server that is hosting the VM, I don't see another way around virtual console access.

I'll update if I come across anything else.

get-esxtop != resxtop

2011-02-06T14:29:00.001-08:00

Bleh. get-esxtop is no alternative for resxtop. You windows users are better off getting vmplayer with vma inside.

ESXi

2011-01-31T13:22:00.000-08:00

To follow up on Call "UserDirectory.RetrieveUserGroups" for object "ha-user-directory" on ESXi "" failed.

YEP! Changing the LDAP SSL certificate requirements from "required" to uh.. not.. made the error go away on our domain.

Computer configuration - Policies - Windows Settings - Security Settings - Local Policies/Security Options - Domain Controller: LDAP server signing requirements (None/Require signing/Undefined[which is the same as None])

A quick google search brought up this likewise discussion, where a member of likewise states that they don't support ldaps.

The ticket for me was the "LDAP error code: 8 (Strong(er) authentication required)" line in /host/messages. No verbose logging was required to get to the root of the problem.
Good enough for me and my crew.

ESXi Active Directory Lookup failure

2011-01-28T16:03:00.000-08:00

Call "UserDirectory.RetrieveUserGroups" for object "ha-user-directory" on ESXi "" failed.

Wha?

ESXi 4.1.0 v320137, evaluation license

Looks like a known bug: http://communities.vmware.com/message/1688839.

Note my message there at the bottom that says that the actual authentication and user/group add/del works fine... you just have to manually type the users/groups.

Here's my traceback in /host/messages:

Jan 29 00:43:07 Hostd: [2011-01-29 00:43:07.905 FFDC2B90 verbose 'UserDirectory' opID=C6A12DE4-00000176] Searching for LDAP server for AD
Jan 29 00:43:07 Hostd: [2011-01-29 00:43:07.909 FFDC2B90 verbose 'UserDirectory' opID=C6A12DE4-00000176] Using LDAP base dn: DC=ad,DC=mycompany,DC=com
Jan 29 00:43:07 Hostd: [2011-01-29 00:43:07.910 FFDC2B90 verbose 'SysCommandPosix' opID=C6A12DE4-00000176] ForkExec '/bin/kinit', pid 30416, rc 0
Jan 29 00:43:07 Hostd: [2011-01-29 00:43:07.959 FFDC2B90 error 'UserDirectory' opID=C6A12DE4-00000176] LDAP error code: 8 (Strong(er) authentication required)
Jan 29 00:43:07 Hostd: [2011-01-29 00:43:07.959 FFDC2B90 error 'App' opID=C6A12DE4-00000176] Error accessing directory: Can't bind to LDAP server for domain: AD
Jan 29 00:43:07 Hostd: [2011-01-29 00:43:07.959 FFDC2B90 info 'App' opID=C6A12DE4-00000176] AdapterServer caught exception: 68130fd8
Jan 29 00:43:07 Hostd: [2011-01-29 00:43:07.960 FFDC2B90 info 'Vmomi' opID=C6A12DE4-00000176] Activation [N5Vmomi10ActivationE:0x68094b10] : Invoke done [retrieveUserGroups] on [vim.UserDirectory:ha-user-directory]
Jan 29 00:43:07 Hostd: [2011-01-29 00:43:07.960 FFDC2B90 verbose 'Vmomi' opID=C6A12DE4-00000176] Arg domain:
Jan 29 00:43:07 Hostd: "AD"
Jan 29 00:43:07 Hostd: [2011-01-29 00:43:07.960 FFDC2B90 verbose 'Vmomi' opID=C6A12DE4-00000176] Arg searchStr:
Jan 29 00:43:07 Hostd: ""
Jan 29 00:43:07 Hostd: [2011-01-29 00:43:07.960 FFDC2B90 verbose 'Vmomi' opID=C6A12DE4-00000176] Arg belongsToGroup:
Jan 29 00:43:07 Hostd: (null)
Jan 29 00:43:07 Hostd: [2011-01-29 00:43:07.960 FFDC2B90 verbose 'Vmomi' opID=C6A12DE4-00000176] Arg belongsToUser:
Jan 29 00:43:07 Hostd: (null)
Jan 29 00:43:07 Hostd: [2011-01-29 00:43:07.960 FFDC2B90 verbose 'Vmomi' opID=C6A12DE4-00000176] Arg exactMatch:
Jan 29 00:43:07 Hostd: false
Jan 29 00:43:07 Hostd: [2011-01-29 00:43:07.960 FFDC2B90 verbose 'Vmomi' opID=C6A12DE4-00000176] Arg findUsers:
Jan 29 00:43:07 Hostd: true
Jan 29 00:43:07 Hostd: [2011-01-29 00:43:07.960 FFDC2B90 verbose 'Vmomi' opID=C6A12DE4-00000176] Arg findGroups:
Jan 29 00:43:07 Hostd: true
Jan 29 00:43:07 Hostd: [2011-01-29 00:43:07.960 FFDC2B90 info 'Vmomi' opID=C6A12DE4-00000176] Throw vmodl.fault.SystemError
Jan 29 00:43:07 Hostd: [2011-01-29 00:43:07.960 FFDC2B90 info 'Vmomi' opID=C6A12DE4-00000176] Result:
Jan 29 00:43:07 Hostd: (vmodl.fault.SystemError) {
Jan 29 00:43:07 Hostd: dynamicType = ,
Jan 29 00:43:07 Hostd: faultCause = (vmodl.MethodFault) null,
Jan 29 00:43:07 Hostd: reason = "Error accessing directory",
Jan 29 00:43:07 Hostd: msg = "",
Jan 29 00:43:07 Hostd: }

Also seems from the logs that there's something running in the background using the credentials that added the host to AD for some other lookup. Not sure how I feel about that one.

I'll see if I can devote any more time to this next week. As it is, it just looks to be an obnoxious bug...

ESXi log file location

2011-01-12T14:41:00.000-08:00

Damned if I had a brain freeze today and couldn't remember the url path where ESXi has it's log files:

https://hostname/host

There.

New VMware Licensing (updated)

2010-12-07T18:07:00.000-08:00

ESX licensing doesn't look to be changing. The quick word back from our rep is:

"We will be changing how some of the management products are licensed (moving from socket-based to vm-based)."

New Cluster Voting

2010-12-07T18:06:00.000-08:00

New vSphere licensing on the way?

2010-12-02T11:43:00.000-08:00

I heard a scoop that VMware has hired the individual that was in charge of Oracle's per core licensing model... So there's a good chance that vSphere will see licensing changes. We'll see if my vmware contacts can confirm anything.

Will Steve drink the kool-aid?

2010-11-09T15:32:00.001-08:00

A co-worker passed on this well written letter to Apple from UW-Madison to support virtualization, especially in a post Xserve world. Drink Me

Which made me remember a bit of research I did for a potential client in Jul 2009. Below is the email in full:

---
I got a question about VMWare's enterprise product line on Apple Hardware (as this is a requirement to run OSX). Looks like there was a recent thread started in a few places online talking about this. Here is the thread, excerpts below. http://communities.vmware.com/thread/213146

From a VMWare employee:
vSphere support for xServe requires supporting UEFI. This capability is on VMware's roadmap for a future release. Sorry, I can't give you an ETA but hope that helps.
Thoughts from me:
2011? 2012? I wouldn't think sooner.

From a VMWare customer:
So, though the Parallels Server Virtualization Product IS more Serverish than running Fusion, it is still not Bare Metal, and definately NOT VSphere.

Info from me on this:
http://www.parallels.com/products/server/mac/features/ "Parallels Server for Mac" requires OSX to be installed on the hardware. At one point in the marketing on their site they call it a "bare-metal" solution but it DOES require a base install of OSX to run the APPLICATION "Parallels Server for Mac".
The single biggest thing that precludes us from looking at other hypervisor solutions is memory optimization. ANY other hypervisor solution (HyperV, Xen, xVM, Parallels, etc..) do not have the ability to oversubscribe memory. When the same piece of physical hardware can comfortably host 5 Hyper-V VMs or 20 ESX VMs I'll choose ESX.

For "illegal" use of vSphere on xServe hardware we've got this jem from that thread:
The Xserves DO NOT have BIOS emulation in the EFI firmware like the desktops do (http://community.brighton.ac.uk/ajd9/weblog/15666.html).
Thoughts from me:
So you could get ESX on a laptop or MacPro but not an XServe.

More thoughts from the peanut gallery:
For an enterprise-grade solution I would want to wait for ESX to run on apple hardware or for Apple to allow OSX to run on ESX legally. Anything else would not have the same level of redundancy and support that comes along with an enterprise-grade price tag.

---

In a post Xserve world, what are Apple enterprise clients left with? Currently nothing.

ThinProvisioning a Solaris VM with ZFS

2010-11-09T14:42:00.000-08:00

We have some clients who like to purchase storage well over what their expected use will be. We've started thin provisioning the data disks on any VMs from these particular clients, or for any new request over 100G. So we've built in some over-provisioning of our storage into the service while watching the capacity at both the current thin and theoretical thick levels.

And then I started watching the disk usage creep on a few VMs: Our Solaris 10 machines that I've been particularly keen to keep the performance and customers happy. Our Sol admins switched from UFS to ZFS root. ZFS and it's partner-in-crime, RAID-Z use Copy-on-write transactional model so that a new block is used upon every write.
So I've asked our Admins to bring the VMs back to UFS or I'd need to thick the disks out.

So there you have it. Every time the Solaris VMs that I'm trying to keep in top shape write something to disk, they're potentially extending the thin provisioned vHDD. VMware would like to think that there's no performance hit with thin disks, but I don't think they've planned for a copy-on-write file system.
The fact that Solaris is officially supported but there's little discussion or documentation has stymied our group's willingness to bring on more Sol in VI. It just looks like VMware and Solaris/Oracle are headed in different directions.

VMWorld 2010 sessions

2010-11-08T19:35:00.000-08:00

VMware requires auth for the root of the vmworld session directories, but not the actual content files.
Using the bash script linked below will download all available flv presentations or the session pdf if no flv was available. Some sessions did not have any content (about a dozen or so). Resulting folder size is 12.7G. Requires lwp-download that should work on sol/nix/OSX ..as long as perl is installed.
Or just browse the script for sessions you want. Session name will be the file name.

...if anyone asks, i got it from a friend of a friend.

Drink Me

Drinking the virtual kool-aid

2010-11-03T13:38:00.000-07:00

Although kool-aid seems to be well established with vmware, virtualization and "the cloud", vKoolAid had not yet been coined. It was about time for me to split off my vSpecific posts from my generic blog, so I thought some nice orange Kool-Aid would do the trick. Tasty orange drink.

Where to get fast EMC information?

2010-11-01T16:38:00.000-07:00

My storage dude passed this gem on to me a little while ago:
"Unfortunately I just received word that all R30(to) upgrades have been temporarily placed on hold until mid-December."
So, hm. I head to powerlink and I don't see any huge OMG warnings. I look at Chad's blog and have assumed correctly that there would be nothing there. Sound like a bug. A big bug. A month-and-a-half-to-figure-out-what's-going-on bug.

And I don't know where to look for fast info from EMC. Finally a use for twitter?

VMWorld alumni "gift" HA!

2010-08-30T14:38:00.000-07:00

I give over 3k to attend VMWorld for multiple years and what is my
"thank you" gift? A pair of $0.50 headphones. How about a VIP bump in
new sessions, you losers. I'm unsure what VMWare wants for this:
http://twitpic.com/2jriaf but I don't think I'm interested.

vSphere performance overview charts working and printing

2009-11-17T15:23:00.000-08:00

I was getting the IE "Navigation to the webpage was canceled" message for the performance overview window after upgrading to vSphere.
Fix was to verify (and change) the URL to a universally resolvable DNS
record in the file:
C:\Program Files\VMware\Infrastructure\VirtualCenter
Server\extensions\com.vmware.vim.stats.report\extension.xml
(thanks to: http://communities.vmware.com/thread/233471).

Next was to see that no, there was no print button. BUT! You can use IE's print function that happens to be available in a few small whitespace areas. Here's one:

The second chart down is Memory (MBps). Right click in the empty whitespace between this graph and the left side of the frame. One of your IE options is print, another is print preview. There may be a spot on the first graph or other areas of the page, but this is a sure spot. Total hack, brought to you by me without google's help.

vmware vMA 4

2009-10-01T14:11:00.000-07:00

It would be nice if vMA would work with virtualcenter 2.5. I was sad
to get an error message today. Looks like direct ESX 3.5 targets (even
if they are managed by a 2.5 virtual center) works.

VMWare Rap

2009-07-30T16:08:00.000-07:00

The obvious winner in a contest (that's still running):
http://www.youtube.com/watch?v=zBQgfWdrUCs

The vid reminded a co-worker of this one (save yourself and click 7 minutes in):
http://video.google.com/videoplay?docid=4915875929930836239

VCP4 beta

2009-07-16T14:43:00.000-07:00

Possible time 4hrs, 15 minutes. Time taken: 3hrs 25 minutes with no
review. 270qs
Things I didn't think would be covered in great detail: NFS, SANboot,
DataRec, maximums
Things that were missing: vZones, other add-ins coming soon (chargeback,etc)

Surprisingly few really stupid questions "On such-an-such screen, what
is your configured ability for xyz? 1/2/4/8?"

Doh! I can't delete my datastore!

2009-05-21T17:07:00.000-07:00

This is for ESX 3.5, not sure if it will work for ESX4.

run on one host:
ls -l /vmfs/volumes
find the datastore ID (by simlink name)
Run on all hosts:
vmware-cmd -l | grep datastoreID

In my case, it was a template that still had reference to the datastore.

There may be a more elegant solution using RCLI so you don't need to
run the command on all your hosts, but this was quick and dirty.

Old Dell hardware fluke

2009-03-04T15:24:00.000-08:00

raises it's ugly head. Updating the PERC controller on a 2850 makes
the local vmfs volume inaccessible, even on 3.5 u3. Luckily the
commands for the workaround from 3.0 are still valid. KB Article
1001577 : http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1001577

Now I thought I updated to the latest PERC firmware..back to the
software update repos...

stupid easy PS vm disk info

2009-02-10T11:51:00.000-08:00

There are lots of fancy ways to get esx disk info from powershell.
Some that I found gave "interestin" results so this is my KISS
version:
Get-VM | Get-Harddisk | export-csv c:\temp\disk.csv

I take out the junk I don't need and then do a data -> text to columns
to parse out the [datastore] vm_name\vmdiskname.vmdk

raw data good.

"By Design"

2009-02-10T11:46:00.000-08:00

To follow up on stumped stumped stumped. There is definitely some funny memory management going on behind the scenes that doesn't show up in the balloon driver or shared memory stats. Here's some email thread, the top is a much smarter colleague giving his guess to the behavior:
----
I have to say that I don't quite get it. Without balloon driver activity, there is no induced memory pressure on a guest. It would have to be...

Ah ha! (maybe). What they're (barely) saying seems to require dynamic behavior from the active memory algorithm. We've talked about how the definition of the "working set" cannot be precisely defined. The active flag for a page could be tied to a decision threshold proportional to the real memory versus granted memory ratio or something similar. So if you populate a 16GB server with only a 1GB guest, any page that was ever used will remain active. As you start adding guests (or allocating guest memory), the threshold lowers, and some pages that were counted as active now expire based on some ranking attribute involving age, frequency, and/or patterns of past use. And all this would happen even if there was memory to spare, because the hypervisor starts preparing for heavier use preemptively.
Good theory?

This isn't what you describe but could explain some of the observed behavior.

Are you saying that newer processors with hardware-assisted VT
http://communities.vmware.com/docs/DOC-9150 don't perform page sharing? Or just that page sharing doesn't really start happening until the guest ratio kicks up? Or that it depends on something we don't understand yet? [[ Depends on something else or guest ratio...
]]

I love this quote from doc 9150: "However, TLB misses are much more expensive in a nested paging environment[*], so workloads that over-subscribe the TLB are potentially still good candidates for binary translation without hardware assistance."

[*] from the AMD-V + RVI feature set which the AMD Opteron 2300s in our r805 have, enabled by "monitor.virtual_mmu = "hardware":
http://www.amd.com/us-en/0,,3715_15781,00.html?redir=SWOP08 has a little more history about support in VMWare than I'd seen.

That is mysterious and ineffable. So hardware VT is sometimes bad but good luck figuring out when. Down the rabbit hole we go.

-A
-----
My email to VMWare support to close the case:

Melori (and other support),
It seems like this might be a wild goose chase. I've been trying to recreate the memory differences yesterday and today and can't do it. I de-populated one of the old servers and it's showing the same memory values on two different sets of cloned guests as the new (fairly empty) servers.
Looks like any server with a high guest ratio will work differently with memory than one with a low number of guests. I'm seeing a metaphor for shoving multiple pillows into a pillow sack or something.
They condense up without loosing their ability to work, but without page sharing (so it seems...the pages shared numbers didn't fluctuate).
So it does seem to be "by design" but more difficult for capacity prediction models to work with.

Stumped stumped stumped

2009-01-12T15:56:00.000-08:00

I've been running around in circles for months now about how our linux
(RHEL5) VMs are gobbling up memory. We've got old(er) intel 64bit dual
dual-core with HT and new shiny AMD 64bit NUMA RVI dual quad-core
machines. Memory usage on the AMD boxen is through the roof, as in
80-90% utilized when left to it's own devices, while the intel stay
around 50-60%. RVI is supposed to save the world by dumping one layer
of memory virtualization. Virtualized MMU (hardware page table
virtualization) should 'just work'. The host should see roughly what
the guest sees.

So how come some guest cron jobs take most available (guest) physical
memory and then shove them into cache? The host still thinks the
memory is needed so it's not readily available for other guests. I ran
across the wonders of drop_caches and an echo 1 brings my physical
memory usage from 450MB to 90MB (and dumps the buffers / caches). The
host still shows the memory in use by the guest (esxtop, vc stats) but
it's not. no swapping, no idea what the hell is going on.

I would love to blame redhat and their shitty setup (come on,
bluetooth running on a minimal install?), but a debian box has a
similar issue. SELinux? Although not as horrible with SELinux off,
this is not the root.

But my real question with allathis is: Why the hell can't I find
anyone else with beef on how this is working? I have google kungfu
skillz and can't find anything that talks about this as a performance
problem (or not a problem).

yech.

VMware Update Manager Gotchas

2008-11-24T15:52:00.000-08:00

We've got a production and development vcenter area that are both on the same subnet. No problems there. We then installed a dev and prod version of UpdateManager. Uh oh. When I go into VIC -> plugins, which one is showing? There's no way to know until you get the ssl cert warning. How do I change or get rid of the connection? By uninstalling VIC, dumping files and reinstalling. From there, make sure the VUM that you don't want isn't turned on. Yuck.
Did you know that the update signatures command is triggered on VirtualCenter not UpdateManager?

VMWare ESX lunreset cliffnotes

2008-06-19T11:26:00.000-07:00

We had a path failover that caused one of the ESX boxes to not be happy with one of the LUNs (shown as zero bytes in VIC).
Cliffnotes as to what the steps were to remedy it. *This worked fine on our environment with other ESX hosts in the cluster actively using this LUN with running windows and linux guests. Posts in the vmware forum have a warning that this could hoze you. I agree to tread lightly with these commands (I called vmware for the below solution, I didn't want my head on the chopping block if an oracle vm crashed).

Find the LUN that is showing 0.00B:

71 esxcfg-vmhbadevs
72 ls /vmfs/devices/disks/

Kick it while watching it's progress:

77 vmkfstools -L lunreset /vmfs/devices/disks/vmhba0:1:1:0
78 tail vmkernel
79 vmkfstools -L targetreset /vmfs/devices/disks/vmhba0:1:1:0
80 tail vmkernel
81 vmkfstools -L busreset /vmfs/devices/disks/vmhba0:1:1:0
82 tail vmkernel

Make sure it's there in one spot:
83 esxcfg-vmhbadevs
84 less vmkernel

rescan to find it in another:
85 esxcfg-rescan vmhba0

In VIC, refresh on storage page (not rescan on storage controllers) to have it
really available to the host.

VMWare ESX 3.5 RHEL5U1

2008-06-12T21:22:00.000-07:00

I moved a guest RHEL 5 update 1 from ESX 3.0.1 to 3.5.1 tonight. It wouldn't boot. Well, it would start to boot, get to init and then shut itself off with no error messages on the guest or host.
Culprit: Host options had the OS still set to RHEL4 (from 3.0.1). Switching it to RHEL5 made the world turn again (after much worrying).