Aria Operations Report Tracking Datastore Over-commitment.

CaptainvOPs

Blog Date: January 16, 2024

One of my customers was interested in tracking datastore over-provisioning in Aria Operations, since they started deploying all of their virtual machines with thin-provisioned disks. After doing some digging, I found there is a Overcommit ratio metric for datastores, so in this blog I will review the creation of a custom view that we will then use in a report.

In Aria Operations under Visualize -> Views, create a new view. In this example, we’ll just call it Datastore Overcommit. Click NEXT

Now we can select the metrics desired. We will want to add the subject of “vCenter | datastore”, and then you could also group by “vCenter|Datastore Cluster” if you desire. For this example, I have selected the following datastore metrics:
Metric: “Summary|Parent vCenter”. Label: “vCenter”
Metric: “Disk Space|Total Capacity (GB)”. Label: “Total Capacity”. Unit: “GB”
Metric: “Disk Space|Total Provisioned Disk Space With Overhead (GB)”. Label: “Provisioned Space”. Unit: “GB”
Metric: “Disk Space|Virtual Machine used (GB)”. Label: “Used by VM”. Unit: “GB”
Metric: “Disk Space|Freespace (GB)”. Label: “Freespace”. Unit: “GB”
Metric: “Summary|Total Number of VMs”. Label: “VM Count”. Unit: “GB”
Metric: “Disk Space|Current Overcommit Ratio”. Label: “Overcommit Ratio”. Sort Order: “Descending” Coloring Above: Yellow Bound: “1”. Orange Bound: “1.3”. Red Bound: “1.5”

The end result should look something like this:

I typically will set the Preview Source as “vSphere World” to see the output data I am getting.

If you don’t like the datastores being grouped by the datastore cluster, then just undo the grouping option, and all of the datastores that are the worst overcommit offenders will rise to the top. The view can now be saved. 

Creating an Aria Operations Report.

In Aria Operations, Under Visualize -> Reports, create a new report. In this example we call it Datastore Overcommitment.

In section 2 for views and dashboards, I searched for datastore and found the newly created “Datastore Overcommit” view created earlier. I dragged it to the right. I changed the Orientation to landscape, and turned on Colorization.

From here, under section 3 you can select the format of the report PDF and/or CSV, and then under section 4 you can elect to add a cover page and what not. I personally like getting a PDF and CSV. Now can click SAVE to save the report. 

From here, you can run the report or schedule it. It’s that simple.

Aria Operations Dashboard: VM Guest File System Usage

CaptainvOPs

December 2023
Aria Operations 8.12.1

For the past couple of months, I have been working with a customer developing Aria Operations (formally vROps) dashboards for various interests. The dashboard I’ll cover here was one I created to help them track and identify the guest file system usage of the virtual machine. This works for both Microsoft and Linux based systems.

Box 1a is a heatmap widget configured as a self provider configured to refresh every 300 seconds. Additional configuration as follows:

The heatmap is a nice visual that will turn red as the guest file system consumes disks on the VM to spot problems. You then select a box in the heatmap to populate the 2a. Box 2a then feeds data into 2b, 2c, 2d, and 2e.

Box 2a is a custom list view widget i created that lists several metrics of the virtual machine with custom metric labels. It is configured to auto select the first row.

Those metrics are:
“Badge|Health%“,
“Configuration|Hardware|Disk Space“,
“Guest File System|Utilization (%)“, (Coloring above: Yellow 75, Orange 80, Red 90);
“Virtual Disk:Aggregate of all instances|Read IOPS“, (Coloring above: Yellow 100, Orange 200, Red 300);
“Virtual Disk:Aggregate of all instances|Write IOPS“, (Coloring above: Yellow 100, Orange 200, Red 300);
“Virtual Disk:Aggregate of all instances|Read Latency (ms)“, (Coloring above: Yellow 10, Orange 20, Red 30);
“Virtual Disk:Aggregate of all instances|Write Latency (ms)“, (Coloring above: Yellow 10, Orange 20, Red 30);
“Datastore:Aggregate of all instances|Total Latency (ms)“,
“Datastore:Aggregate of all instances|Total Throughput“,
“Disk Space|Snapshot|Age (Days)“, (Coloring above: Yellow 7, Orange 14, Red 21);
“Disk Space|Snapshot Space“.

Box 2b is a Scoreboard widget configured to list the selected VM details regarding information on how the VM is configured.

Configuration is set like so:

Under Input Transformation, set to self.

Output Data will be configured as follows:

Box 2c is a metric chart widget with the Input Transformation configured as self, and the Output data configured to use the virtual machine metric “Guest File System|Utilization”.

Box 2d is simply the Object Relationship widget.

Box 2e is another custom list view and is configured to refresh every 300 seconds. 

This list view is configured to do an instance breakdown of the following metrics:

“Guest File System:/|Partition Utilization (%)“, (Coloring above: Yellow 75, Orange 85, Red 95);
“Guest File System:/|Partition Utilization“;
“Guest File System:/|Partition Capacity (GB)“;
“Capacity Analytics Generated|Time Remaining“.

Box 3a is fed data from 2e so that we can see how the virtual machine disks are behaving on the datastore(s).

This is another custom list view configured as follows:

Configuration is set to refresh content at 300 seconds. Output data is configured with a custom list view with the following metrics:
“Devices:Aggregate of all instances|Read Latency (ms)“, (Coloring above: Yellow 10, Orange 20, Red 30);
“Devices:Aggregate of all instances|Write Latency (ms)“, (Coloring above: Yellow 10, Orange 20, Red 30);
“Devices:Aggregate of all instances|Read IOPS“, (Coloring above: Yellow 100, Orange 200, Red 300);
“Devices:Aggregate of all instances|Write IOPS“, (Coloring above: Yellow 100, Orange 200, Red 300);
“Devices:Aggregate of all instances|Read Throughput“;
“Devices:Aggregate of all instances|Write Throughput“.

Those are all of the configured widgets on this dashboard. The integration schema will look like this:

I do hope to share this dashboard with the VMware Code sample exchange, and I will update this blog once that has been completed. I hope my breadcrumbs above will enable you to create a similar dashboard in the meantime.

VMware Cloud Foundation: SDDC Manager Day 2 Configurations: Deploying an Application Virtual Network In The Management Domain.

CaptainvOPs

Blog Date: 10/20/2023

In the previous blog, I went through the process of deploying an NSX edge cluster via VCF in the management domain. In this blog, I will walk through the process of deploying an application virtual network (AVN) for the management domain for the vRealize/Aria suite of products. This deployment assumes that we will be deploying the latter in a VCF aware configuration which would be typical for most deployments. AVNs allow the SDDC Administrator to configure vRealize/Aria management applications for software defined networking through NSX-T Data Center.  AVNs configure a local region and a cross-region SDN segment providing security, mobility and flexibility of vRealize/aria management applications. vRealize/aria components can be moved between regions to maintain operations during planned migration, maintenance or in the case of a DR event.

For this, click workload domains.

Click on the management domain.

Prior to deploying AVNs, an NSX Edge cluster of two (2) or more nodes is required. In my previous blog, I walked through the SDDC Manager automation that is used to deploy the edge cluster. This post assumes the edge nodes and cluster are in a healthy state, but you can always look via the SDDC manager by clicking on the Edge Clusters tab.

Assuming the edge cluster and nodes are healthy, click Actions and then Add AVNs.

Select Overlay-backed NSX segment and click NEXT.

In the NSX Edge Cluster drop menu, select the management edge cluster. In the NSX Tier-1 Gateway drop menu, select the gateway. Click NEXT.

Fill out the network specs for Region-A. An MTU of 9000 was used here to keep the MTU consistent in the environment.

Fill out the network specs for the X-Region section. An MTU of 9000 was used here to keep the MTU consistent in the environment. Click VALIDATE SETTINGS, and then click NEXT after the validation succeeds. Otherwise, remedy the errors and validate the config again.

Review the configurations for accuracy, and click FINISH.

Watch the tasks window in the SDDC manager for deployment task to succeed.

You can also click the main task to see all of the sub tasks and watch for them to successfully complete.

Going back to the summary tab of the management domain, below the NSX-T configuration, you will now see the details of the AVN network just deployed.

Now we have the AVN ready for the vRealize/Aria suite deployment through the SDDC manager.

For more information, see VMware’s documentation Deploying Application Virtual Networks in VMware Cloud Foundation.

VMware Cloud Foundation: SDDC Manager Day 2 Configurations: Deploying an NSX Edge Cluster In The Management Domain.

CaptainvOPs

Blog Date: 10/13/2023

After VCF has been deployed, and the tier 0 and tier 1 gateways have been configured in NSX, you can log into the SDDC manager and configure the NSX Edge Cluster for the management domain/consolidated domain deployment. A similar process is also used for the NSX Edge Cluster in a workload domain. During the design preparations for the VCF deployment, the NSX cluster would have been defined, along with the network information. Those design considerations and work wont be covered here, and it is assumed you are ready to deploy the edge cluster.

In the left pane of the SDDC Manager, select Workload Domains, and then click the management domain link. In the right pane, click the ACTIONS drop-down menu and click Add Edge Cluster.

Review the prerequisites, select the Select All check box, and click BEGIN.

Provide the following information for the new edge cluster. In this example, we are using an MTU of 9000 for simplicity.
Edge Cluster Name:
MTU: 9000
Tier 0 Name:
Tier 1 Name:
Edge Cluster Profile Type: Default

Enter the desired password for the password boxes. Click NEXT. Enter the details for what the edge cluster will be used for. On the Edge Cluster Settings, in this example we will select Custom.

The edge cluster size will be determined during the design phase of the NSX edge cluster. In this example, we have Large selected. Your mileage may vary.

Validate that the Tier0 Service High Availability is Active-Active. Validate that EBGP is selected as the Tier0 Routing Type. Enter the correct ASN number for your environment. Click NEXT.

Provide the cluster details for the first edge node. Cluster type setting will vary depending on your environment configuration:

Fill in the Edge Node details for the TEPs.

Now add the first Tier-0 uplink and BGP info, and second Tier-0 uplink and BGP info.

Review the values entered to insure accuracy, and then click the ADD EDGE NODE button. The config for the first edge node has been saved.

Now you need to add the config details for the second edge node. Click ADD MORE EDGE NODES which takes you to the top of the Edge Node Details page. You’ll notice the previous values entered are still present. Starting from the top, work your way to the bottom, and carefully update all entries with the second edge node config.

Review the values entered to insure accuracy for the second edge node, and then click the ADD EDGE NODE button. The config has been saved.

At this point, two edge nodes should be shown. More can be added if needed, but two nodes offer sufficient redundancy. Click NEXT.

Review the summary and ensure that you entered the values correctly.

IMPORTANT:

Ensure that you entered the IP addresses, FQDNs, and VLANs correctly by comparing all values with the network diagram. Any data entry error will cause errors in deployment.

Click NEXT

Wait for validation to complete.

If any checks fail, record the cause, and verify that no values are incorrectly entered in the wizard. CAUTION: Do not click FINISH unless the validation succeeds for all checks. If the validation checks succeed, click FINISH.

You can monitor the deployment in the SDDC manager tasks pane in the bottom of the screen. If you click on the task name for adding the edge cluster, the sub-deployment tasks will be shown.

Wait for the edge cluster deployment task to complete before deploying application virtual networks.

For more information, see VMware’s documentation on Managing NSX Edge Clusters in VMware Cloud Foundation.

VMware Cloud Foundation: SDDC Manager Day 1 Configurations: Configure Certificate Authority

CaptainvOPs

Blog Date: 10/6/2023

After VCF has been deployed, you can log into the SDDC manager and configure a certificate authority. This is useful if there’s a desire to use custom signed certificates. SDDC manager has the ability to run OpenSSL and create certificates for the associated VMware products in the SDDC. Alternatively, you can also configure the Certificate Authority to use a Microsoft CA. In this blog I will cover the steps to setup the OpenSSL. One thing to note is that by default, the certificates authored using the method will be valid for a year.

In the left pane, scroll down and click Certificate Authority under Security. In the right pane, click the EDIT button. In the Certificate Authority Type, select OpenSSL. Fill in your environment specific details, and then click SAVE.

The CA Configuration Saved Successfully message appears on the page.

For more information, checkout VMware’s documentation on Managing Certificates in VMware Cloud Foundation.

To update the certificates in SDDC manager, I would first take a snapshot of whatever you are updating the certificate of, then in SDDC manager locate Workload Domains in the left pane under Inventory. Select the management domain (or desired domain), and then click on the Certificates tab.

In this example, place a check mark in the box for sddcmanager, and click the GENERATE CSRS button to configure the CSR details with your organization’s specific details.

Click NEXT.

On the Subject Alternative window, you may wish to add additional info, otherwise click NEXT.

Click the GENERATE CSRS button. Wait for the CSR Generation is successful message before continuing. You might need to refresh the browser periodically.

Now we can generate the signed certificate from the certificate authority, which in this case will be the SDDC manager considering we enabled OpenSSL. Leave sddcmanager selected, and click GENERATE SIGNED CERTIFICATES button. In the Generate Certificates pop-up window, click the Select Certificate Authority drop-down menu and select OpenSSL. Click GENERATE CERTIFICATES. You will need to wait until you see the certificate generation is successful message. View the Certificate Operation Status column to see that the certificate generation was successful.

With the sddcmanager still selected, click the INSTALL CERTIFICATES button. Refresh the browser a few times until you see security warnings about the new certificate as a result of the change. In the end, you should see a Certificate Installation is successful message display.

At this point you can restart the SDDC manager appliance to ensure the new certificate is in use going forward. Now simply rinse, wash, and repeat on the remaining appliances you wish to update the certificate.

In my next blog, I’ll go over the process of deploying an NSX edge cluster in the management domain. this will be used in the future when we create the network segment for vRealize/Aria.

VMware Cloud Foundation: SDDC Manager Day 1 Configurations: Connect to an Online Depot

CaptainvOPs

Blog Date: 9/29/2023

Another day 1 operation in a newly deployed VCF environment, would be to configure a service account to connect the SDDC manager to your organization’s VMware Customer Connect account.

This service account would need the following permissions: Access to view licenses, products, and be able to download products. This service account would need to be configured in the VMware Customer Connect portal. For the user email address, I typically have customers use a group distribution email. Configuring SDDC manager to use a service account, will allow it to access the VMware Depot without relying on a employee account, which will be more secure. Just be sure to set a strong 18-20 character password, and record the account details in a secure location. *PRO TIP: Log into the customer connect portal using the service account you created, and verify that you can view licenses, products, and download products. This will insure the account meets all access requirements that the SDDC manager will need when we configure the depot settings below.

Just like we did in the previous blog, we’ll click the question mark in the upper right, and select Guided Setup.

On the next screen, we’ll want to continue with Step 2 Configure SDDC Manager. Click VIEW DETAILS.

Select Connect to the Online Depot and click NEXT.

If there’s no proxy to configure for the SDDC manager to reach the internet, click SKIP CONFIGURE PROXY.

Add the details for the VMware depot service account mentioned above and click AUTHORIZE.

Assuming the service account has the correct permissions, you shouldn’t get any errors. A connectivity error may indicate a firewall access issue.

To validate that you can see and download bundles, under Lifecycle Management in the left hand menu, select Bundle Management. (No available bundles when screenshot was taken).

For more information on how to connect to the VMware Depot, see VMware’s Documentation Download and Install Bundle from SDDC Manager.

In the next blog, I will cover how to configure a certificate authority in the SDDC manager using Open SSL.

VMware Cloud Foundation: SDDC Manager Day 1 Configurations: Identity Provider

CaptainvOPs

Blog Date: 9/22/2023

Another day 1 operation in a newly deployed VCF environment, would be to configure an identity provider. Just like we did in the previous blog, we’ll click the question mark in the upper right, and select Guided Setup.

On the next screen, we’ll want to continue with Step 2 Configure SDDC Manager. Click VIEW DETAILS.

In the Configure SDDC Manager wizard, click Connect Identity Provider, then click Next.

Here, we will be configuring the identity provider that will be used in the vCenter. Click the Select Identity Provider from the drop-down menu, select Embedded. Click the Select Identity Source drop-down menu, select AD over LDAP. Click Next.

Fill in the LDAP Settings.

In this example, I don’t use a certificate for LDAP connectivity. Your mileage may vary. Click NEXT. Validate the information on the Review page matches the table above, then click SUBMIT. Wait for the save to complete, then validate that Connect an Identity provider has a green check mark.

Now that a identity provider has been established, you can now configure access to those who you trust to perform administrative tasks. The SDDC manager already comes preconfigured with a vsphere.local group called sddcadmins defined in the vSphere. Depending on your security needs, you can also add users and groups directly in the SDDC manager. In this example. I will configure access through the sddcadmins group in vSphere.

If we drop down into the vSphere, first we will want to set the identity provider as the default authentication source.

Next, you’ll want to locate the SDDCAdmins group and add the organization’s trusted admins who will be administering the SDDC. Typically, I have customers define an AD group with the SDDC admins, so that you only need to define the group in the vSphere. As the AD account membership to the group organically changes with users, you won’t have to worry about updating the vCenter group.

Likewise, you can also add the vSphere admins group defined in AD to the Administrators group in vSphere.

For more information on configuring access in the SDDC, see VMware’s documentation on Managing Users and Groups in VMware Cloud Foundation.

VMware Cloud Foundation: SDDC Manager Day 1 Configurations: Backups

CaptainvOPs

Blog Date: 9/15/2023

After VCF has been deployed, you can log into the SDDC manager and configure the backups for the SDDC manager and NSX. (This does not configure vCenter backups, so you will still need to log into the vcenter:5480 to configure that manually). You can opt to use the guided setup by clicking the question mark in the upper left.

When the wizard loads, On the left side, click Register a Backup Server and click NEXT.

Select the four prerequisite check boxes and click NEXT. Register the external SFTP server to enable automatic backups for SDDC Manager. Note: The SSH fingerprint text box is automatically filled once a successful connection to the backup server has been made.

Click Register. Wait for the registration to complete and the Setup Automatic Backup page displays. Setup Automatic Backups configure the wizard with the values that best suite your backup strategy.

Click Save.

After the backup settings have been saved, it is wise to manually kick-off the backup job by clicking Backup Now. You can monitor the backup task job status in the bottom tasks pane.

For more information, see the official VMware documentation -> Backup and Restore of VMware Cloud Foundation.

In the next blog, I’ll walk through connecting VCF to an identity provider.

See you there! We’ll be Presenting: DRaaS and On-Premises Disaster Recovery Solutions at The 2023 US VMware Explore in the CXS Theater!

CaptainvOPs

It’s that time of year again, and I am very excited that VMware Explore has been moved back to sunny Las Vegas! To that end, I am also very excited to announce that I get to co-present a CSX Theater session CXS1280LV this year with Jack Levy.

VMware customers interested in leveraging new or existing cloud investments for Disaster Recovery and Ransomware Recovery, will most certainly be interested in attending our 45 minute presentation at the CSX Theater in the solutions exchange area.

A link to the CXS Theater session in the content catalog can be found here: DRaaS and On-Premises Disaster Recovery Solutions: Go from Zero to Hero

In today’s world, disaster recovery is not only about protecting the enterprise during a data center failure. The truth is there are bad actors out there also trying to extort money through ransomware. It’s time to get serious about disaster recovery, and VMware is here to help. Protect your business from human, climate and ransomware disasters. Learn from our experts about VMware’s various disaster recovery solutions from on-premises to our disaster recovery-as-a-service (DRaaS) offerings with ransomware recovery options in the cloud. Define the recovery strategy and protect your assets. In this session, we’ll be covering VMware Site Recovery Manager with vSphere replication, VMware Cloud Disaster Recovery plus the optional Ransomware recovery capabilities, and we’ll have a couple brief recorded demos of the setup process.

We will also be giving an exclusive presentation for our TAM and VMware Success 360 customers in a session that won’t be found in the content catalog TAM3792LV.

I’ll admit that I have always been excited to attend VMware Explore when I was a customer, back when it was called vmworld. My first time as a customer, was back in 2016 when vmworld was moved to Las Vegas. This year, the new Explore will be in Las Vegas for the first time, and it will be my first time presenting as a VMware employee. Does that mean I have come full circle?

Regardless, I am looking forward to the event, being able to meet with my customers I have worked with over the years as a consultant, and getting to meet some of my VMware colleagues that I have had the pleasure to work with over the years.

Jack Levy and I started around the same time on the same team Americas Professional Services Org, and became fast friends.

We are both excited about the opportunity to present at VMware Explore, and we look forward to seeing you at our session!

vCenter 8 Machine SSL Certificate Management

CaptainvOPs

vSphere 8
Windows Server 2019 Certificate Authority
Blog Date: December 16, 2022

Replacing the machine SSL certificate is a breeze in vSphere 7 and 8. Many organizations have security requirements and need for the vSphere web interface to have that secure padlock icon. Most organizations I have come across have a Microsoft Certificate Authority in house, but there are exceptions. In my lab, I have a Microsoft CA configured, and that is what I’ll be using in this example.

VMware has a pretty good KB2097936 article that discusses how to use vSphere Certificate manager.

Installing and managing custom signed certificates is not difficult. Can this be done during normal business hours? I would argue, yes. Some organizations will choose to do it after hours, however. When the certificate is installed, services are restarted on the vCenter to reflect the change. During that minute or two, you won’t be able to manage your vSphere infrastructure from the vSphere client, but all of your ESXi hosts, and all of your workloads will continue to run untouched. If the certificate is bad, you can always revert to snapshot and have everything as it was prior to replacing the SSL certificate in a matter of minutes. Again, the ESXi and VMs will be unaffected. However, what you do need to keep in mind, is that when you replace the certificate on the vCenter, other services that direct connect to the vCenter like Aria (vRealize) Operations, log insight, automation, network insight, etc., would all need to have their connection to the vCenter refreshed. Any third party solutions out there would also need to have their connection to the vCenter refreshed.

  1. I recommend that you take a snapshot prior to messing with certificates.
  2. After you’ve logged into the vSphere client with an admin level account, click the three hash marks next ‘vSphere Client’ in the upper left (some call it a hamburger), and select “Administration” from the menu.
  3. On the Administration page, select “Certificate Management” under ‘Certificates’.
  4. On the Certificate Management screen, you will see Trusted Root Certificate at the bottom and Machine SSL Certificate at the top. In this example, we are only worried about the Machine SSL Certificate. Just below it, you will see an “Actions” drop menu, and from the menu we need to select Generate Certificate Signing Request (CSR).

5. Fill out the specific details for your certificate. Every box should be filled out. Boxes marked optional are just that. I personally fill out the Subject Alternative Name with the FQDN and IP address on the vCenter comma separated. Click NEXT.

6. You can COPY or DOWNLOAD the CSR. In this example, click COPY and then click finish.

7. Now we need to connect to the Microsoft CA web portal. Click “Request a Certificate”.

8. Click advanced certificate request

9. Paste the vCenter CSR created earlier into the saved request box. Next, chose the Certificate Template. I created a VMware template using the VMware knowledge base article here KB2112009.

10. Select “Base 64 encoded”, and then click “Download certificate”.

11. Now that we have downloaded the base 64 encoded certificate for the vCenter, we also need the CA certificate. We can download this from the Microsoft CA Web Portal homepage. Click “Download a CA certificate”

12. For the Encoding Method, select Base 64, then click “Download CA certificate”.

13. In the vSphere client Certificate Management screen, click the Actions drop menu and select “Import and Replace Certificate”.

14. On the Replace vCenter Server Certificate screen, select the “Replace with external CA certificate where CSR is generated from the vCenter Server….” option and click NEXT.

15. Upload the vCenter Machine SSL Certificate in the top box, and the CA certificate in the bottom box. In some configurations, your organization might have more than one CA, so you may need to upload a chain CA root certificate(s). In this example, I only have one CA certificate, so there’s only one to upload. Once you click REPLACE, know that the vCenter will become unavailable while services are restarted.

16. You will either need to dump your existing cookies for the vSphere environment that you just replaced the certificate for, or use private/incognito browser mode and reconnect to the vCenter. Services should restart in a few moments. As long as your certificate was correct, you should now see the secure padlock.

17. Once logged into the vCenter, you can go back to the Certificate Management screen, and you should see the new Machine SSL Certificate. You will also see a new Trusted root certificate has been installed. From this point forward, you will need to manage both the Machine SSL certificate and the CA Trusted Root Certificate that has been installed on the vCenter. Depending on organizational policies, the validity length of these certificates will differ. In my home lab, I have my Microsoft CA configured for 10 years because I know I will rebuild it prior to the certificates expiring, and I don’t want to manage certificates more than I have to.

When everything checks out, don’t forget to clean your room and delete the snapshot.