Accessing the VMware ESXi Hidden Console

I have lately installed the free version of the VMware ESX server, namely VMware ESXi Server. I was taken a back seat when I was told that the ESXi edition does not have a service console, which is good, but there are times when you need to perform some troubleshooting. There are many benefits not having the service console – less overhead, fewer patches, and greater security. With ESXi, the “console” is a simple yellow and black menu driven text interface with only the most basic options. However, lately I found out that ESXi actually has an extremely thin linux-based console that can be accessed.

With the full version of VMware ESX Server, that has been available for years, there is a special “virtual machine” that runs a modified version of Red Hat Linux Enterprise. That special VM is called the service console and is used to administer the ESX host system.

Officially, VMware says that you should administer your ESXi server using either the VI Client or the CLI VMware RCLI. Thus, if you want to perform commands and scripting on your ESXi server, you need to install the remote command line interface on your Windows PC.

However, there is another CLI interface for ESXi that can be used to run commands directly on the server. This is in contrast to RCLI where the command is run on your local management PC and connects to the ESXi host over the network. The difference is that with RCLI, you cannot, say, edit a remote file as you could do if you were using the traditional ESX Server service console.

Thus, the only way to edit a file like /etc/hosts or /etc/inetd.conf is to access this hidden & unsupported thin linux interface and edit these files with vi. Also, with the ESXi hidden console, you can run commands like esxtop, esxcfg-route, and vmkfstools. Isn't this great!

How do I access the VMware ESXi hidden Console?

Accessing the hidden & unsupported ESXi console is not difficult if you know how to do it. However, if you do not know how to do it, there is no menu option or easily accessed help file that tells you how to access it.

To access the hidden & unsupported ESXi console, you must go to the console of the server. You cannot access this console via RCLI, RDP, the VI client, or other method. The only way to access the ESXi console is to go to the console of the server.

Once you are on the server’s console, press Alt-F1.

At that point you will see a console log of what has happened on the server but there is no prompt and no help file available. If you type something, it will not appear on the screen.

What you need to do is to type the command unsupported and press enter. his will not appear on the screen. When you do this, here is what you will see:

This activates what VMware called “Tech Support Mode”. As the warning says, this is unsupported unless you are using it with help from VMware Tech Support. Because of this, neither VMware nor I can make any warranties if, by using this interface, something unexpected happens to your ESXi Server. Because of that, you should only do this on a TEST system.

Now, type your ESXi Server root password.

At this point, you are successfully logged into the hidden ESXi console. So what can do you once you are in here? Let’s find out…

What can I do inside the VMware ESXi hidden console?

The ESXi hidden / unsupported console is a “Linux-like” interface but extremely light when compared to a real Linux installation, which I believe is the proprietary VMware OS. For example, some of the most basic Linux commands work like ls (to list files), cd (to change directories), rm (to remove files), cp (to copy files), vi (to edit files), and reboot.

However, other common Linux commands do not work, such as more, pg, nano, or man.

The most interesting configuration files are located in /etc, just like in Linux. The most useful commands that you can execute are located in /sbin.

In this article, I am offering a quick overview of the ESXi command line but for a more complete reference you should read chapter 2 of the VMware Remote Command-Line Interface Installation and Reference Guide because that covers the vicfg-xxxx commands in detail. However, inside the ESXi console, you run most of those same vicfg-xxxx commands but they start with esxcfg-xxxx instead (the deprecated version). In fact, the RCLI Reference Guide (link above) has a chart that shows the esxcfg-xxxx to vicfg-xxxx equivalent command syntax.

In my opinion, the most important thing that I have used the hidden/unsupported ESXi console for is to edit text configuration files on the ESXi Server. This is important because, as I said, this cannot be done using the RCLI. For example, here are a few of the text files I have edited:

1. /etc/hosts – due to issues related to ESXi servers coming and going randomly from my VMHA resource pool, a VMware Tech had me edit the /etc/hosts file to statically make host entries for the other ESXi hosts in the RP. This was done to rule out any DNS issues.
2. /etc/inetd.conf – this file can be used to enable services that, otherwise could not be enabled. For example, by removing the hash (#) mark before the ssh or the ftp, I can enable these services on my ESXi server.

Here is an example of editing the inetd.conf file to enable SSH:

Of course, there are other files that can be edited or viewed, like the passwd file or inittab.

Again, I offer the warning that all of this is unsupported by VMware unless you are performing these steps under their direction.

Conclusion

In this article, I covered the "hidden" & unsupported VMware ESXi Server console. Almost everyone knows that ESXi doesn't have a service console but it does have a hidden console. In this article, I also demonstrated the benefit of using this hidden console. Primarily, that benefit is that you can edit text files directly on the server to allow you to enable services like SSH. However, as you have access to the server’s console, and can do much more than you could in the simple console menu interface, the possibilities of tweaking and configuration are only limited by the limited command set on the server.

"Power Supply Failure Brings Down HP BladeSystem c7000" HP admits

A power supply failure in HP BladeSystem c7000 enclosures can cause the whole BladeSystem to fail, the firm has admitted.

According to an HP advisory note: "HP has identified a potential, yet extremely rare issue with HP BladeSystem c7000 Enclosure 2250W Hot-Plug Power Supplies manufactured prior to March 20, 2008. This issue is extremely rare; however, if it does occur, the power supply may fail and this may result in the unplanned shutdown of the enclosure, despite redundancy, and the enclosure may become inoperable."

So, the issue is extremely rare, says HP. But it applies to any HP BladeSystem c7000 Enclosure configured with an HP c7000 Power Supply, if the power supply was manufactured before March 20, 2008. Each enclosure can have up to a total of six power supplies.

My understanding is that all the power supplies in the enclosure are connected together, forming a single power domain. The blades in the system connect to a single power bus. If the power supply fails then all the blades may stop working meaning that all their applications, including any virtual machines, go offline. Effectively, there is a single point of failure and redundancy limitation in the HP’s BladeSystem c7000 design.

HP's advisory goes on to say that: "BladeSystem c7000 Enclosure Power Supplies manufactured on or after March 20, 2008, and DC-powered enclosures (typically utilized in an Integrity blade environment) are not affected. To ensure stability of your computing environment, HP is providing a power supply identification utility to enable customers to identify potentially affected power supplies. Supplies identified by the utility will be replaced by HP."

Defective power supplies will be replaced free of charge. HP provided a statement about the issue: "HP has been made aware of a very small number of incidents involving power supply failures in the BladeSystem c7000 enclosure. Because customer service and product quality are top priorities for HP, the company is working with HP BladeSystem customers to replace all potentially affected c7000 power supplies purchased by customers."

The All New Polished Vista, Introducing "The Windows Mista"

What's the excitement behind Windows 7, do you remember this:

Ballmer: Windows 7 is Vista, just 'a lot better' [link]

"Windows Vista is good, Windows 7 is Windows Vista with clean-up in user interface [and] improvements in performance," Ballmer said. [link]

The event helped me to crystallize my thoughts. So to crystalline it... is it an UPGRADE or an UPDATE??  Microsoft prefers to call it an MAJOR UPGRADE to SELL MORE and ofcourse to get people move on from the Windows XP to Vista. For Vista users its can be called a MINOR UPGRADE or even better it should be an UPDATE or to put it more accurately a service pack. 

For Microsoft its a fairly significant upgrade, but for Vista users its not an overhaul of the operating system rather a signaficant update. But lets put a question to Mr. Ballmer, are they planning to sell Windows 7 to UPGRADERS and allow it for download as a service pack for UPDATERS??

If you dont understand, I cant explain it to you any better....the OS was not built for the user, it was built for Microsoft to make more $$ at your expense.....get used to paying for the true definition of "slack" ware... should be called Microsoft Mista. They Missed-another chance to do the right thing and make it secure.

Its just the same chocolate with a new packaging and a new brand name? 

"Polish doesn't change quartz into a diamond"

VMware Consolidated Backup Design Preparation and Understanding for Backup Administrators

While I am working on designing a Virtual Infrastructure Solution, I thought of penning down a few lessons learned for my future reference as well for consultants who are planning to design a similar solution. Backup is one important area to be considered. One of the advantages of purchasing VMware Infrastructure Enterprise (VI 3.5) is that along with the flagship ESX hypervisor there are additional licensed features and products included that are necessary to create business continuity for virtual machines (VMs). VMware Consolidated Backup (VCB) is one of these products. Often misunderstood as the complete answer for a virtual data center, VCB requires some preparation and understanding for backup administrators currently used to the traditional physical enterprise backup solution.

VCB is not the entire backup solution for virtual infrastructure
It is very rare that VCB allows administrators to completely remove all backup agents from virtualized servers. This is because VMware Consolidated Backup does not:

• Perform specialized application backups (like Microsoft Exchange Information Store or Windows Server System State)
• Perform file-level backups of non-Windows VMs
• Provide management, cataloging or archiving of backup files
• Provide direct file restores to virtual machines
• VCB is a framework of scripts that needs to be integrated with a third-party backup application to provide these features.

VCB should be installed on a dedicated Windows Server
It is recommended VCB be installed on its own server. Also known as the VCB Proxy Server, this system has the following requirements:

• Microsoft Windows Server 2003 Service Pack 1 (32‐bit or 64‐bit) or higher
• Media repository managed by the third-party backup application's management server
• The same storage protocol access as the ESX hosts to the VMFS LUNs where the VMs are stored. (i.e., host bus adapters (HBAs) for access to Fiber Channel storage or initiator configuration for iSCSI storage). Depending on the version of Windows Server used, automatic partition mounting will have to be disabled before attaching the VCB server to the VMFS LUNs
• Dedicated disk storage for the VCB Holding Tank where backup and restore files are written
• Third-party backup agent

VCB needs a large disk volume for a Holding Tank
Along with the shared access to the ESX LUNs, VCB also needs a large disk volume formatted as NTFS, which will become the Holding Tank for backup images. This volume can be on the SAN or the local VCB server's disks. The Holding Tank volume is where full VM images are placed both during backups and restores.

Therefore, the size of the Holding Tank is critical in the design. For example, if a virtual infrastructure consists of VMs that take up 1 TB of disk space and the expectation is that a full VM backup is to be taken nightly, then the Holding Tank volume needs to be large enough to support 1 TB of backups. Another scenario would be to alternate groups of full VM backups in order to decrease the required size of the volume. In this case, administrators still need to make sure the Holding Tank is large enough to hold the VM using the most disk space.


The role of the third-party backup agent
The third-party backup application does the actual backing up and management of the files. Once VCB copies a VM image to the Holding Tank it is then up to the third-party backup application to move those files to whatever media repository is in use. It is also the function of the agent to clear out the Holding Tank so that the next scheduled job has available disk space to complete.

In the case of file-level backups, VCB also mounts the copied VM image (in thumb drive style as previously mentioned) so that the backup agent can see the VM's file system. The backup agent can then perform full, incremental or differential file-level backups to the media repository. In some scenarios, the single agent on the VCB server can replace the multiple agents on the VMs.
VMware maintains a compatibility guide for supported third-party backup applications. Many of these supported applications have VCB integration modules that coordinate the scheduling of the VCB scripts and the agent backup from within the application's GUI.


Understanding VCB restore jobs
Restoring files leverages the third-party backup agent's ability to move files from the media repository back to the Holding Tank. Once the VM image is back, it can be copied in full to a VMFS volume or mounted like a thumb drive again so that individual files can be restored. An administrator must manually copy files to the restore location in both scenarios.
VMware Converter, most often used to migrate physical servers to virtual machines, can also create VMs from VCB images. Therefore, VMware Converter can be a more effective full VM restore tool in some cases. Check out VMware's Virtual Machine Backup Guide for more detailed information on implementing VCB.