Reset GRUB/root Password for vCenter/PSC Appliance

In Redhat/Fedora/Cent, GRUB can be protected by running the grub-md5-crypt command and pasting the outputted password hash into the grub.conf file. vSphere 6.0 password protects grub by default. If you change the root password in the VAMI, the GRUB password is changed to match. If you do not change the root password, the GRUB password is "vmware".

To reset the GRUB password, we need to boot into a Cent or Redhat live CD. The ISO can be obtained here: https://www.centos.org/download/. Its best to upload the ISO to a datastore that the appliance has access to.

Stop the appliance and attach the ISO:

Be sure to select the "Connect at Power On" option. Boot the VM into the ISO and select the "Troubleshooting" option. 

Next, choose "Rescure a Red hat (or CentOS depending on your ISO) Enterprise Linux System"

Select “Continue” to mount the VCSA 6.0’s  root filesystem in Read/write mode under /mnt/sysimage.  RHEL 7.2 is capable to detect the VCSA’s root volume and mounts it.

The VCSA root filesystem is mounted under /mnt/sysimage and you can now access (and modify) it using the shell. Navigate to /mnt/sysimage/boot and list the contents. You'll see we now have access to the grub directory:

cd to the grub directory and list the contents. Look for a file called "menu.lst". This file holds the grub boot loader password. Open this file with vi by typing "vi menu.lst". Navigate to the line beginning with "password" using the arrow keys, and then type "dd" to remove the line. 

You can then save the file by pressing ":wq" (without quotes). You can now cat the file and see that the password has been removed. 

Exit the shell (this will reboot the server). Detach the ISO and boot the appliance. Once the system is booted, stop the VCSA in the GRUB menu (by pressing the escape key during boot) to break the OS root password. 

Press "e" to edit the boot commands for the kernel. 

Append "init=/bin/bash" to the line in this step and press enter.

Press "b" to boot the system. 

You will now boot into a bash shell where you can set the root password.

Once this is done, exit the shell by typing "exit". You can now boot the appliance and login with your new root password. 

Could not connect to VMware Directory Service via LDAP when Deploying New vCenter Appliance

 

Problem:

Deploying a brand new vCSA 6.7 appliance results in the following error during the second stage of the deployment.

 

 

Solution:

This problem is almost always caused by DNS resolution. Once you create the appropriate A and PTR record for your appliance on a LOCAL DNS server, you should be to successfully complete the deployment. Local DNS resolution is required, you cannot use a public DNS server while installing vCenter. For example, 8.8.8.8 will not work.

Since you have already completed Stage 1 of the deployment, you can login to the appliance via SSH and update the DNS settings. This will only work if you chose to enable SSH during Stage 2 of the deployment. 

SSH to the appliance and run "/opt/vmware/share/vami/vami_config_net" (without quotes). Choose option 4 to update DNS settings and option 3 to update the hostname (if necessary). The deployment wizard states that a hostname is optional, but it is actually required. I have never had a successful deployment without specifying the hostname.  

 

You can then verify the DNS settings have been updated in the resolve.conf:

 

Azure Site Recovery - VMware-to-Azure: Wrong IP address discovered for VM

When replicating virtual machines from VMware to Azure using Site Recovery, you may encounter an issue where the Configuration server discovers the wrong IP address for a VM. This can be caused by stale entries within the infrastructurevms MySQL table that is used by ASR to track VM attributes. 

To resolve this issue, you first need to disable replication for the VM in the Azure Portal.

Next, login to your ASR Configuration Server and open a CMD prompt as administrator. Browse to the bin directory for your ASR installation. For example, in my case ASR is installed on the E: partition under the following directory:

E:\Program Files (x86)\Microsoft Azure Site Recovery\home\svsystems\bin

Type in this command to remove the VM from the ASR database (replace IP address with the IP of your VM):

perl Unregister-ASRComponent.pl -IPAddress 10.0.0.4 -Component Source

That’s it. You should now be able to reconfigure replication for the VM, and ASR will discover the correct info about the VM.

Remove Stubborn PSC or vCenter Appliance from an SSO Domain

While attempting to decommission one of our vCenter sites, I ran into an issue removing one of the PSCs. This site consisted of two PSCs and one vCenter appliance. I removed the first PSC from the SSO domain successfully, and then removed the vCenter appliance. Things became a little tricky during the removal of the final PSC. This PSC did not get removed even after running the cmsso-util command. This article will detail the steps I took in decommissioning the site, as well as removing the stubborn PSC.

First, we need to check if vCenter is currently using the PSC we plan on decommissioning. If it is, we need to use the cmsso-util command to redirect vCenter to a different PSC. Instructions for redirecting vCenter can be found here: https://kb.vmware.com/s/article/2113917?language=en_US

To check what PSC your vCSA is currently pointing to, browse to the Advanced Settings for the vCSA in the vSphere Web Client. Filter by this key: config.vpxd.sso.admin.uri

To remove a PSC or vCSA from an SSO domain, connect to a PSC via SSH and run these commands:

To remove a PSC from the vSphere SSO domain:

cmsso-util unregister --node-pnid psc01.ad.vcplab.local --username administrator@your_domain_name --passwd vCenter_Single_Sign_On_password

To remove a vCSA from the vSphere SSO domain:

cmsso-util unregister --node-pnid vcsa.ad.vcplab.local --username administrator@your_domain_name --passwd vCenter_Single_Sign_On_password

After running these commands, delete the virtual appliances. You can also verify the appliances have been removed by browsing to Administration > System Configuration > Nodes in the vSphere Web Client.

If cmsso-util fails to remove any of the nodes, you can use this command to force the removal:

vdcleavefed -h vcsa.ad.vcplab.local -u Administrator -w Passw0rd!

Upon successful completion, you should see something like this:

/usr/lib/vmware-vmdir/bin/vdcleavefed -h vcsa.ad.vcplab.local -u administrator
password:
vdcleavefd offline for server vcsa.ad.vcplab.local
Leave federation cleanup done

When specifying the username, use the SSO admin (administrator). However, do not use the full UPN (administrator@vsphere.local). Doing so will cause the command to fail.

Enable SNMP in ESXi 6.5 via SSH

I was trying to enable SNMP on a few stand-alone ESXi 6.5 hosts the other day. Unfortunately, when attempting to enable this service using the vSphere Host Client, I received a very unhelpful error:

 



I wasn't able to find any helpful information in the logs that could help identify the cause. Interestingly enough, enabling the service through SSH worked. Here's how I did it. First, you'll need to connect to your host via SSH (you will need to enable the SSH daemon first).

Once connected, type in "esxcli system snmp get" to see the current snmp configuration.



To enable SNMP, we first need to configure the community strings. To do that, type in "esxcli system snmp get --communities=<community strings>". Then, enable the service by typing in "esxcli snmp set --enable=true".

You can then verify the new configure by typing in "esxcli system snmp get" again.



Now refresh the vSphere Host Client, and the service should be running.

vSphere - Unable to Apply DRS Resource Setting on Host

I noticed this error on one of my hosts today. After reading the full error message on the Events tab, and paying particular attention to the "Error Stack" section, it was obvious what the problem was.



"The configuration file of the virtual machine is corrupted". Well, that helps narrow things down, but what virtual machine is it?

After clicking on the host and going to the Related Objects -> Virtual Machines tab, the problem became even more clear.



This was a replica that we deleted last weekend. For whatever reason, vCenter didn't de-register it from the database when it was deleted. The VM no longer existed on the host. To fix the issue, just right click on the VM and click "Remove from Inventory".

After doing this DRS became healthy and all alarms were silenced. I hope this helps anyone experiencing this or a similar problem.

-Ryan

Unable to Power on Virtual Machine - A General System Error Occurred: Connection Refused

Hi there,

Earlier today I cloned a VM in one of our vCenter appliances. The cloning process completed without a hitch (as it usually does). However, I was not able to power on the VM. I looked at the "Tasks" view in vCenter and found this:



This is a fairly generic error message. However, the "Connection Refused" part prompted me to take a look at the vCenter services. You can view these services by browsing to Home > Administration > System Configuration > Services in the vSphere Web Client for vCenter. On the Summary tab, there is a handy "Services Health" section that will give you a high level overview of the overall health of the services. You can see here that three of the services are in a critical state on my vCenter appliance.



You can hover your mouse pointer over the critical services link and view the critical services. You can also click on each service to get a better view of the status. The Postgres database service on my appliance was one of the services in critical state. I clicked on it and viewed the screen below. You can see here that the problem is obvious, the filesystem holding the Postgres database service is completely out of space. I was able to identify that the log directory was the culprit.



Its simple enough to SSH into the appliance and delete the logs. However, this will not prevent the problem from happening again. After doing some Googling (I love that the Merriam-Webster dictionary identifies this as a real word in the English language!), I found these KB articles:

Follow this one to decrease the maximum backup size and maximum backup index size of the SSO logs in the log4j.properties configuration file:

/storage/log directory is full in vCenter Server Appliance 6.0 (2143565)

To offload your logs to a syslog server, follow this article:

Redirect vCenter Server Appliance Log Files to Another Machine

This will prevent the log directory from consuming the entire partition again. After following these steps I rebooted the vCSA for good measure and was able to power on the virtual machine after doing so.

Hopefully this will help someone who is having the same (or similar) problem!

Running vSphere in VMware Workstation 12

In this post I'll be walking through how to run a vSphere lab in VMware Workstation. I recently decided to obtain VCP6-DCV. Rather than driving up my electric bill like I've done in the past using physical servers, I'm attempting to run the entire lab on my workstation and a Synology NAS. 

If you've ever installed ESXi, installing it in Workstation will be a familiar process for you. VMware tools is included in the installation disc, which makes installing ESXi in Workstation dramatically easier than it used to be. . The process is very simple, so I won't be going through those steps here unless someone asks me to in the comments. I also will not be going through the process of installing Windows Server or configuring a domain controller/DNS/DHCP, as I am sure you have done so in the past if you are reading this. 

So that really only leaves us with installing vCenter. Most of the blogs I found for installing vCenter in VMware Workstation 12 were not accurate, and often left me with a broken installation. The process is somewhat straight-forward when deploying from the OVA. Let's get started.

First, download the OVA for vCenter here: Download vCenter

Once the download has completed, click File > Open in Workstation. Browse to the OVA, then give your new VM a name and location if necessary. Accept the EULA when prompted. 

Be sure to read it! 😎

Once the OVA finishes importing, do not power on the VM! There is some customization we need to do first. Close Workstation if it is open. Browse to the location on your PC that you imported the VM to. I'm using a Windows OS, so I will use File Explorer. Open the .VMX file (use Notepad or another text editor):

This is the configuration file for your virtual machine. We can use it to customize the name, IPv4/6 details, DNS domain, etc. Scroll down to the last line of text, and paste this in:

guestinfo.cis.vmdir.password = "vmware!"
guestinfo.cis.appliance.net.addr.family = "ipv4"
guestinfo.cis.appliance.net.addr = "10.0.0.15"
guestinfo.cis.appliance.net.prefix = "24"
guestinfo.cis.appliance.net.mode = "static"
guestinfo.cis.appliance.net.dns.servers = "10.0.0.10"
guestinfo.cis.appliance.net.gateway = "10.0.0.1"
guestinfo.cis.appliance.root.passwd = "vmware!"

Customize the above code to your needs. You will likely need to change the IPv4 details. Save the .VMX file and close your text editor. Now you can power on the virtual machine, and vCenter will run through the installation process. The installation can take around 10-15 minute in my experience. You may see generic login screens during the installation of Photon, do not login or interrupt the installation. Once it is complete, you should see the DCUI below:

You should now be able to browse to the IP address or DNS name of your vCenter server. Once you complete the configuration, you can login and see the page below:

In my lab I am running 3 ESXi hosts, 1 Windows Server, and one vCenter server. Plenty to study for the VCP lab. 

Good luck and be sure to leave a comment if you have any questions!