July 2024 – CaptainvOPS

Blog Date: July 11, 2024

In VMware Cloud Foundation 4.5.1, managing certificates of the Aria Suite LCM, NSX, VXRAIL, and vCenter Certificates should be done via the SDDC manager, so that it trusts the components certificate. The official documentation on how to do it can be found here -> Manage Certificates in a VMware Cloud Foundation.

In some cases however, certificates can be replaced/updated outside of the SDDC manager either due to a lack of understanding, or in emergency situations where certificates expired. In either of those situations, the certificate must be imported into the trusted root store on the SDDC manager appliance to re-establish trust to those components. Otherwise, SDDC manager will not function as intended.

Official knowledge base article can be found here -> How to add/delete Custom CA Certificates to SDDC Manager and Common Services trust stores.

The following steps can be used to update the SDDC Manager trust store with the new NSX certificate.

IMPORTANT: Take a snapshot of the SDDC Manager virtual machine. **Don’t Skip This Step**
Use a file transfer utility to copy the new NSX certificate file to the /tmp directory on the SDDC Manager.
Establish an SSH connection to the SDDC Manager as the VCF user, and then issue the su – command to switch to the root user.
Obtain the trusted certificates key by issuing the following command:

cat /etc/vmware/vcf/commonsvcs/trusted_certificates.key

Note: You will see output similar to the following:

p_03ZjNI7S^B7V@8a+
Next, Issue a command similar to the following to import the new NSX-T certificate into the SDDC Manager trust store:

keytool -importcert -alias <aliasname> -file <certificate file> -keystore /etc/vmware/vcf/commonsvcs/trusted_certificates.store --storepass <trust store key>

Notes:
- Type yes when prompted to trust the certificate.
- Enter something meaningful, like sddc-mgmt-nsx for the <aliasname> value.
- Replace <certificate file> with the full path to the certificate file that was uploaded in Step 2.
- Replace <trust store key> with the trusted certificates key value returned in Step 4.
Issue a command similar to the following to import the new NSX-T certificate into the java trust store. Here the storepass is changeit:

keytool -importcert -alias <aliasname> -file <certificate file> -keystore /etc/alternatives/jre/lib/security/cacerts --storepass changeit

Notes:
- Type yes when prompted to trust the certificate.
- Replace <aliasname> with the meaningful name chosen in Step 5.
- Replace <certificate file> with the full path to the certificate file that was uploaded in Step 2.
Issue a command similar to the following to verify that the new NSX-T certificate has been added to the SDDC Manager trust store:

keytool -list -v -keystore /etc/vmware/vcf/commonsvcs/trusted_certificates.store -storepass <trust store key>

Note:
- Replace <trust store key> with the trusted certificates key value returned in Step 4.
Issue the following command to restart the SDDC Manager services:

/opt/vmware/vcf/operationsmanager/scripts/cli/sddcmanager_restart_services.sh
(Optional): You can utilize the SDDC manager SOS utility to check the health of the newly imported NSX-T certificate with the following command:

/opt/vmware/sddc-support/sos --certificate-health --domain-name ALL

Tip:
For more information on the sos utility, check out the documentation here: -> SoS Utility Options (vmware.com)
If everything checks out, remove the snapshot that was taken prior to starting this procedure.

Blog Date: July 10, 2024

To get straight to the punch, this is not supported yet. If you already have an existing VCF deployment, there currently is no supported way to migrate your workload domains to vLCM, per the support article here -> Transition from vSphere Lifecycle Manager Baselines to vSphere Lifecycle Manager Images is not supported with VMware Cloud Foundation. While you can technically use the vCenter UI / APIs to make the switch, it will cause workflows in SDDC manager to break, VMware support/engineering will have to get involved, and most likely the changes will have to be reverted.

If you are in the beginnings of deploying a new workload domain, by default it will use vSphere Lifecycle Manager baselines as the update method, unless you specifically checked “Manage clusters in this workload domain using baselines (deprecated)” during the workload domain deployment. However, this option would require you to have an existing vLCM image prior to the workload domain being deployed. If you don’t have a vLCM image yet, the VMware documentation suggests that you deploy the workload domain using legacy images (VUM), and that documentation can be found here -> Deploy a VI Workload Domain Using the SDDC Manager UI.

Doing a little research on the available options if no vLCM baseline image is available, and if you already have identical ESXi hosts deployed to the VCF environment, in vSphere, you can create a new empty compute cluster, select the option to manage the cluster with vLCM baselines, select a identical host already deployed to the environment to import and create the vLCM baseline from including the NSX driver. Now you have a vLCM baseline you can use for new workload domains and clusters using identical hosts. The new vLCM baseline can be imported into the SDDC manager. One might ask if it is safe to create a new compute cluster using the vSphere UI in a VCF deployment? For this purpose because it is temporary, the answer is yes. Technically, if you add additional compute clusters in vSphere without going through the SDDC manager, the SDDC will have no knowledge of it and won’t interact with it, so for our purposes, it is safe to create the empty compute cluster to create the new baseline, and then just delete the empty cluster when finished. Always remember to clean your room.

Although it will take a little work on the font end if you currently do not have vLCM baseline images to deploy a new workload domain, the above process can be used to create it. Eric Gray put together an excellent blog and YouTube video on this here -> Updating VCF Workload Domains deployed using vLCM Images. This walks us through the process of creating a new vLCM baseline image for a vLCM enabled workload domain to upgrade it, but the same process can be used to create a new vLCM image for a new workload domain with identical hardware.

If you have just deployed a workload domain and selected Manage clusters in this workload domain using baselines (deprecated) (legacy VUM), there is no way to convert it to vLCM baselines (at the time of writing this blog). You have to REDEPLOY the workload domain. You could however, take the opportunity using the above method to create a vLCM baseline image for the workload domain, so that when you redeploy it, you’ll have a vLCM image to use. Silver lining?

Unconfirmed reports indicate that the functionality to migrate existing workload domains from legacy VUM to vSphere Lifecycle Manager baselines is *targeted* for VMware Cloud Foundation 9.