Stumped stumped stumped

I've been running around in circles for months now about how our linux
(RHEL5) VMs are gobbling up memory. We've got old(er) intel 64bit dual
dual-core with HT and new shiny AMD 64bit NUMA RVI dual quad-core
machines. Memory usage on the AMD boxen is through the roof, as in
80-90% utilized when left to it's own devices, while the intel stay
around 50-60%. RVI is supposed to save the world by dumping one layer
of memory virtualization. Virtualized MMU (hardware page table
virtualization) should 'just work'. The host should see roughly what
the guest sees.

So how come some guest cron jobs take most available (guest) physical
memory and then shove them into cache? The host still thinks the
memory is needed so it's not readily available for other guests. I ran
across the wonders of drop_caches and an echo 1 brings my physical
memory usage from 450MB to 90MB (and dumps the buffers / caches). The
host still shows the memory in use by the guest (esxtop, vc stats) but
it's not. no swapping, no idea what the hell is going on.

I would love to blame redhat and their shitty setup (come on,
bluetooth running on a minimal install?), but a debian box has a
similar issue. SELinux? Although not as horrible with SELinux off,
this is not the root.

But my real question with allathis is: Why the hell can't I find
anyone else with beef on how this is working? I have google kungfu
skillz and can't find anything that talks about this as a performance
problem (or not a problem).

yech.

VMware Update Manager Gotchas

We've got a production and development vcenter area that are both on the same subnet. No problems there. We then installed a dev and prod version of UpdateManager. Uh oh. When I go into VIC -> plugins, which one is showing? There's no way to know until you get the ssl cert warning. How do I change or get rid of the connection? By uninstalling VIC, dumping files and reinstalling. From there, make sure the VUM that you don't want isn't turned on. Yuck.
Did you know that the update signatures command is triggered on VirtualCenter not UpdateManager?

VMWare ESX lunreset cliffnotes

We had a path failover that caused one of the ESX boxes to not be happy with one of the LUNs (shown as zero bytes in VIC).
Cliffnotes as to what the steps were to remedy it. *This worked fine on our environment with other ESX hosts in the cluster actively using this LUN with running windows and linux guests. Posts in the vmware forum have a warning that this could hoze you. I agree to tread lightly with these commands (I called vmware for the below solution, I didn't want my head on the chopping block if an oracle vm crashed).

Find the LUN that is showing 0.00B:

    71  esxcfg-vmhbadevs
  72  ls /vmfs/devices/disks/

Kick it while watching it's progress:

77  vmkfstools -L lunreset /vmfs/devices/disks/vmhba0:1:1:0
  78  tail vmkernel
  79  vmkfstools -L targetreset /vmfs/devices/disks/vmhba0:1:1:0
  80  tail vmkernel
  81  vmkfstools -L busreset /vmfs/devices/disks/vmhba0:1:1:0
  82  tail vmkernel

Make sure it's there in one spot:
  83  esxcfg-vmhbadevs
  84  less vmkernel

rescan to find it in another:
  85  esxcfg-rescan vmhba0

In VIC, refresh on storage page (not rescan on storage controllers) to have it
really available to the host.

VMWare ESX 3.5 RHEL5U1

I moved a guest RHEL 5 update 1 from ESX 3.0.1 to 3.5.1 tonight. It wouldn't boot. Well, it would start to boot, get to init and then shut itself off with no error messages on the guest or host.
Culprit: Host options had the OS still set to RHEL4 (from 3.0.1). Switching it to RHEL5 made the world turn again (after much worrying).