I was trying to run ESX 3.0.3 U1 inside of my vSphere 4.0 implementation (so that I could test upgrade strategies to 4.1).
I set the OS as RHEL 4 (64bit) and specified "Use Intel VT-x/AMD-V" under the CPU/MMU Virtualization Options area (rather than automatic). It seemed to boot fine at that point.
Note that the install went fine without poking in the settings area, but it was just hanging at "vmkernel has been loaded successfully" rather than actually booting into the OS.
Now I can get to work.