I recently had a customer engagement with Log Insight, and not only did they want to use it as the main log collector for their infrastructure, but they also wanted it to collect logs from their Windows virtual servers. Good news! There is a content pack called “Microsoft – Windows” that should be installed. This contains a configuration template for windows servers. This is used to create a group, so that every time a windows box has the agent installed, the agent picks up the settings from the server and forwards its logs. Otherwise, we must configure each agent ini file manually which is not ideal. There is also a Linux content pack on the market place that can be setup as well. This blog will focus on the Windows content pack, but the steps for Linux is very similar.
The ” Microsoft – Windows” content pack can be found in the Marketplace in Log Insight located on the Content Packs tab.
Once this is installed, go back to the Administration tab, and click on agents in the left column. Click the down carrot next to All Agents, and find “Microsoft – Windows” in the list. To the right of it, click the double box icon to copy the template.
Name the group, and click copy.
Now you configure the filter to find the windows server. In my example, I chose “OS” “Matches” “Microsoft Windows Server 2016 Datacenter”. Click Save New Group button below.
Now that we have a Windows group defined for the agents, go ahead and install the agent on the Windows Server, and now it will have a proper configuration and begin forwarding its logs. If the box already has the agent installed, you might need to restart the agent, or reinstall it.
Likewise, there is also a Linux content pack on the market place that can be setup as well. For either, we don’t have to create one group to rule them all. You can get creative with your group filters, and have specific groups for specific server functions.
Blog Date: 3/03/2021 Versions Tested: vCSA: 7.0.1 ESXi: 7.0.1 Microsoft CA Server: 2016
Recently I had a customer that wanted to install their custom certificates on a new vCenter, and have it act as an Intermediate CA to install approved certificates on their hosts. This is common, but is something I have never done myself, so I ended up testing this out in my lab. In this long blog post, I will walk through:
Generating a Certificate Signing Request (CSR) for the vCenter.
Signing the request, creating the certificate using a standalone Microsoft CA.
Creating a trusted root chain certificate.
Installing the custom signed machine SSL certificate.
Installing the custom signed VMCA root certificate.
Renew host certificates and test.
Before we get started, it is worthwhile to note if you were unaware that there are different certificate modes in vSphere 7. Bob Plankers does a great job explaining these modes in the following VMware Blog: vSphere 7 – Certificate Management
I have done my best to focus the steps involved, and provided those below while testing this out in my home lab. Your mileage may vary. _________________________________________________________________________________________________________
Snapshot the vCenter
We will be applying Custom Machine and Trusted Root certificates to the vCenter, and it is good practice to take a snapshot of the vCenter appliance beforehand.
We will be using WinSCP to transfer the files to and from the vCenter. You should validate that you can connect to yours first. If you are having problems connecting to the vCenter win WinSCP, follow this KB2107727.
Start an SSH session with the vCenter appliance, and enter “/usr/lib/vmware-vmca/bin/certificate-manager” to launch the utility. We want option #2
Enter “Y” at the prompt: Do you wish to generate all certificates using configuration file : Option[Y/N] ? :
Enter privileged vCenter credentials. The default email@example.com will do the trick. Hit Enter key, then input the password and hit the enter key again.
If there is a valid certtool.cfg file that exists with your organizations information you can select N here.
IF there is no certool.cfg file or you would like to look at the file, use option “Y” here.
In the next series of prompts, you will be required to enter information specific to your environment. The inputs in purple below are examples I used for my lab, and yours will be different: value for ‘Country’ [Default value : US] : US value for ‘Name’ [Default value : CA]: cb01-m-vcsa01.cptvops-lab.net value for ‘Organization’ [Default value : VMware]: CaptainvOPS value for ‘OrgUnit’ [Default value : VMware Engineering]: CaptainvOPS Lab value for ‘State’ [Default value : California]: Colorado value for ‘Locality’ [Default value : Palo Alto] Fort Collins value for ‘IPAddress’ (Provide comma separated values for multiple IP addresses) [optional]: 10.0.1.28 value for ‘Email’ [Default value : firstname.lastname@example.org] : email@example.com value for ‘Hostname’ (Provide comma separated values for multiple Hostname entries) [Enter valid Fully Qualified Domain Name(FQDN), For Example : example.domain.com] : cb01-m-vcsa01.cptvops-lab.net Enter proper value for VMCA ‘Name’ : cb01-m-vcsa01.cptvops-lab.net
Enter option “1” to Generate Certificate signing requests
It will prompt for an export directory. In this example I used /tmp/
Now using WinSCP, copy the files off to the workstation you will be using to connect the the Microsoft CA.
For this next part we will need the root certificate from the Microsoft CA. Right-click on the CA and select “Properties”
On the General tab, click the “View Certificate” button. Then on the new Certificate window that opens, click the details tab and then the “Copy to File…” button
When the wizard starts, click Next.
Select “Base-64-encoded X.509 (.CER), and Click Next
Just like before, browse to a locate to save the cert, and give it a meaningful name. In this example, I just use “root-ca”
Click Next and then click Finish. Close the windows.
In order to create a trusted root, we need to create a new file that has the root chain. This file will include the vmca.cer first, and then the root-ca.cer beneath it. I just open the vmca.cer file and root-ca.cer file with Notepad ++
Save it as a new file. In this example I use: vmca_rootchain.cer.
Using WinSCP, I create a new directory in the vCenter’s root directory called CertStore, and then copy the “vmca.cer”, “vmca_rootchain.cer”, and the “vmca_issued_key.key” there.
Open an SSH session to the vCenter, launch the certificate-manager: “/usr/lib/vmware-vmca/bin/certificate-manager”. First we will replace the Machine SSL certificate, so select option 1
Again we are prompted for vCenter authoritative credentials, and just like before we’ll use the firstname.lastname@example.org account and password.
Now we reach the menu where we will use Option 2 to import custom certificates. Hit enter.
It will prompt for the directory containing the cert file for the Machine SSL. In this example I used: /CertStore/vmca.cer
Now it prompts for the custom key file. In this example I used: /CertStore/vmca_issued_key.key
Now it will ask for the signing certificate of the Machine SSL certificate. Here we will use the custom vmca_rootchain.cer file we created earlier: /CertStore/vmca_rootchain.cer
Respond with “Y” to the prompt: You are going to replace Machine SSL cert using custom cert Continue operation : Option[Y/N] ? :
Assuming all of the information in the files was correct, the utility will begin replacing the machine SSL certificate. Watch the console for any errors and troupleshoot accordingly. 9 out of 10 times fat fingers are the cause for errors. Or dumb keyboards..
After clearing the browser cache, we can see the secure padlock on the login page for vSphere 7.
Now log into vSphere, go to Administration, and Certificate Management. Depending on how quickly you logged in after applying the certs, you may see an error on this page. Give it up to 5 minutes, and you’ll see a prompt at the top of the screen asking you to reload. Do it. Now you will see a clean Certificates Management screen. You should see a _MACHINE_CERT with your organization’s details. You will also see three Trusted Root Certificates. One of which will have your Organizations details.
If you have hosts already attached to the vCenter, and you would like to “Renew” the certificate, by default you will need to wait 24 hours before the certificate can be updated. Otherwise you will receive a similar error in vCenter:
If you need to update the certificate right away, follow KB2123386 to change the setting from 1440 to 10. This will allow for updating the ESXi certificate right away. I’d personally change it back after.
Either 24 hours has passed, or you followed the KB2123386 mentioned above. Now you can “Renew” the certificate on the ESXi host(s). You’ll see the Issuer of the certificate has changed, and should reflect your information. This operation doesn’t appear to work while the host is in maintenance mode.
If you browse to the ESXi host, it too will now have a padlock on its login page signifying a valid certificate.
My time with IntegraTouch LLC has been nothing short of amazing. Through this role as a Senior consultant, I have been able to work along side customers in different public and private industries. I have been challenged professionally, mentally, and technically. I have earned the respect from my customers as a trusted advisor, an educator, and trainer. I have taken the time to enjoy the little things in life, like the rush of getting to board a plane and jetting off to a customer site in a different city/town, instead of the normal workday at the same office day after day. I have nothing but nice things to say about my time working for David, Kevin, and the IntegraTouch LLC family. For the first time in a long time, I felt like I could actually breathe in this role and make my own decisions, and I really feel it helped me grow professionally. While working for IntegraTouch, I found my voice to go speak at many different VMUG UserCons, which is saying a lot considering the social anxiety I feel that comes with being an introvert. But, continuously meeting new customers and starting new assignments, is a lot like that nervous feeling we all get starting a new job. I think consulting, and being a public speaker has helped me in many ways.
I’m still honestly in disbelief where I have found myself these past couple of months. I remember growing up, my parents always encouraged me to do better. I grew up in rural New York, where farm tractors, horses and cows, were more common than traffic jams. This life I’ve chosen has been an uphill battle, but I’ve been able to surround myself with friends and family who support me, encouraged me, forever driving me forward. I can look back on all that I have accomplished, and how far I have come today. My video game hobby fostered a relationship with technology that I still hold close today. It has been a rough winding road, but it is with great excitement that I can finally announce that I will be joining VMware as a full time Senior Consultant, SDDC – on January 11th! Working for VMware as a sub contractor has been fun these past two years, but working for the company as an employee has been a career goal of mine since 2015. A career milestone has just been acquired, and I am looking forward to what the future will bring.
Second – validate hardware, firmware, and drivers of ESXi hosts.
Third – validate compatibility of partner products deployed in the environment.
Snapshot vCenter Before
I personally like taking a snapshot of the vCenter VM. I do this after a clean shutdown.
Prerequisites Updates patch the current vCenter appliance (vCSA). An upgrade replaces the existing appliance, so a few extra items will be needed: – Validate adequate storage space is available. – Current vCSA VM will need to be renamed in the inventory, else the new vCSA VM will need a temp name. – Temp IP address will be needed briefly for the new appliance being deployed (assuming Static temp IP being used).
Upgrade The vCenter
Mount the ISO to a jump server Locate and launch the installer.exe.
You’ll be presented with several available options. In this example I am upgrading an existing vCSA, so that’s the option I’ll select.
Enter the details for the existing vCenter Server Appliance. Hit CONNECT TO SOURCE.
This will open up a new wizard which will require environment specific connection details. You can either target a specif host to deploy the new vCenter Server Appliance to, or you can use the vCenter. Sometimes DNS may get in your way and the deployment will fail. Try again using IP addresses instead.
Here you have the option to select a new deployment size based on your needs and future expected growth.
Here you can elect to use a TEMP static IP address or DHCP for the new vCenter Server Appliance being deployed. This will enable it to coexist with the current vCSA while the data and settings are copied over. After a successful deployment, the current vCSA will be shutdown, and the newly deployed appliance will assume the olds address. Make sure you select the appropriate network. This trips more people up than one would think.
Review your settings, and if everything looks good click FINISH.
This may take a while depending on the size of your environment, but the new appliance will now be deployed. Eventually the wizard will prompt to CONTINUE.
System checks will now run to ensure proper configuration and communication.
My lab threw a couple cautions, but nothing was a show stopper, and I was able to continue the upgrade.
This is where the data will be copied from the existing vCenter Appliance. Pick what needs to be copied over and click NEXT.
Review the config and click FINISH if everything is satisfactory.
IF everything was successful, the new vCenter Server 7.0U1 appliance should now be in command, and the original will be powered off. Next steps will be to import your new license keys for 7, and get the new vCenter licensed. In the next post I’ll walk through the steps of upgrading ESXi to 7 using the newly rebranded Lifecycle Manager that replaced VUM.
This is a continuation of deploying VMware Cloud Director 10. In my last post, I walked through configuring the vSphere lookup service, and adding the vCenter (here). In this post I’ll go over adding a Provider Virtual Data Center (PVDC).
Adding a PVDC
Log into the vCD provider interface, and switch to the Cloud Resources view by clicking the menu to the right of vCloud Director logo. Select the Provider VDCs option in the menu on the left, and then “NEW” link to begin.
On page 1, you’ll have to fill in some general information about the PVDC. Give it a name and description meaningful to the resources the PVDC will be connected to. In this example, I am connecting to my home lab. Click NEXT.
On page 2, select the vCenter and click NEXT.
On page 3, you’ll see the available resources. This would be for both compute and storage. In this example I am using a lab, so I only have one available. Hardware compatibility is also configured here for the future tenants deployed to this PVDC. Click NEXT
On Page 4, the available storage policies configured in the vCenter that the tenants would use in this PVDC, will be available for selection here. Click NEXT.
On Page 5, your mileage may vary depending how your environment is configured. In my lab example, I have chosen the default selection. Click NEXT.
On page 6, you are presented with a confirmation of the selected config. Make any adjustments, and click FINISH.
Be patient as it can take some time to build the PVDC. Just monitor the recent tasks for task progress and completion. The end result should show a “Normal” for a status under the configured Provider VDCs.
At this point, the provider side configuration is almost complete. We still need to configure the public facing address. If this were a production deployment, we also find it necessary to configure a VIP/load balancer to place in front of the VCD appliances to handle traffic load. For production deployments there would also be the need to setup signed certificates for the appliances.
In my next blog I’ll go over configuring the public address.
This is a continuation of deploying VMware Cloud Director 10. In my last post, I walked through deploying additional appliances for database high availability (here). Today we will add the vSphere lookup service and the vCenter.
Configuring the vSphere Lookup Service
Log into the vCD provider interface, and switch to the Administrator view by clicking the menu to the right of vCloud Director logo. Then under Settings on the right select vSphere Services.
After clicking the “REGISTER” link in the upper right, we’ll be able to add the lookup service URL link for the vCenter we’ll connect to a little later. Registering with vSphere Lookup Service enables vCloud Director to discover other services, and allows other services to discover this instance of vCloud Director.
Once all of your information is added, click the Register button. Watch the task for successful completion below.
Adding the vCenter
Click on the menu again to the right of vCloud Director logo, and select the vSphere Resources. On the top menu option to the right “vCenters”, click the Add button.
On page 1 of the wizard, fill in the required information for the vCenter. We already configured the lookup service, so we can leave that option selected here to provide the URL. Click next.
On page 2, add the information for the NSX-V appliance. Click Next.
Click finish, and then wait for the vCenter to connect and show a status of Normal.
Now that we have the vCenter connected, we can proceed to setting up and configuring the Provider Virtual Data Center (PVDC). This is required in order to make the vCenter resources such as compute and storage available to tenants. I’ll go over configuring the PVDC in the next blog.
Blog date: September 01, 2020 Completed on vSphere version: 6.7
Today my customer needed to change the root password for roughly 36 hosts across two data centers. These are two data centers that were recently built as part of my residency with them, and they have already seen the benefits of using host profiles. Today I was able to show them one more.
VMware has a KB68079 that details the process should the root password become unknown on a host. Well the same process can be applied and used to update the password on all hosts with that host profile attached. At the time of writing this article, all hosts are compliant with the current host profile, and there are no outstanding issues.
In the vSphere client, go to ‘Policies and Profiles’ and select ‘Host Profiles’ in the left column, click and select the desired host profile on the right.
Edit the desired host profile.
In the search field, type root and hit enter.
Select root in the left column.
In the right column, change the field below ‘Password’ to Fixed password Configuration.
Now you are prompted with password fields and can update the root password.
Click Save once the new password has been entered.
Now you can remediate the hosts against the updated host profile, and the root account will get updated on each host. – Out of an abundance of caution, it is always good to spot check a handful of hosts to validate the new password.
I had my customer go back and edit the host profile once more and change the ‘Password’ field back to: Leave password unchanged for the default account. Click save, and then remediate the cluster again. The new password will stay.
Before I connected with the customer today, they had already researched how to update the root password on all hosts with a script, but this method is simple, automated and built into vSphere.
Quiet often when I connect to customer sites to conduct a health check, I sometimes find their hosts having different NTP settings, or not having NTP configured at all. Probably one of the easiest one-liners I keep in my virtual Rolodex of powerCLI one-liners, is the ability to check NTP settings of all hosts in the environment.
With this rather lengthy command, I can get everything that is important to me.
We can see from the output that I have a single host in my Dev-Cluster that does not have NTP configured. Quiet often I find customers that have mis-configured NTP settings, do not make use of host profiles that can catch and address issues like this.
If you also wanted to see the incoming, outgoing and protocols settings, you could use the following:
Most likely you would have multiple corporate NTP servers you’d need to point to, and that is easily done by separating them with a comma. An example of having two: Instead of having just ‘pool.ntp.org’ I’d have ‘ntp-server01,ntp-server02’.
The next thing needed is the startup policy. VMware has three different options to choose from. on = Start and stop with host, automatic = start and stop with port usage, and off = start and stop manually. In my lab I have the policy set to on.
With that in mind, the following command I can make the NTP settings of all hosts consistent. This command assumes that I only have one NTP server. I am also stopping and starting the NTP service. It is also worth mentioning that each host that already has the ntp server ‘pool.ntp.org’, will throw a red error that the NtpServer already exists.
Since I’ve been on this Professional Services path, as a sub contractor for VMware, I’ve normally been pretty busy. In fact, the majority of last year I spent on the road visiting customers, and so my Monday through Thursday was spent away from home. This made it incredibly difficult to find the motivation to continue my education. This year I’m making use of all the home time during the pandemic to level up. I really love working from home!
In May 2020 I was able to re-certify and pass the VMware Data Center Virtualization Exam for 2020. I spent weeks on prep, and it paid off as I scored 357 out of 300. This was also my first certification at home, rather than going to a testing center.
In July 2020, I decided to go after my first VMware Specialist class certification – Cloud Provider 2020. This certification centers around VMware Cloud Director, and its associated components VMware Cloud Availability and NSX. I’m no longer in the VMware Cloud Provider space, however that’s really where I cut my teeth on VMware technologies, and spent 2014 to 2018 designing, deploying, managing and upgrading five different vCloud Director based cloud environments. I’ve been told that this is a rare skill in VMware’s Professional Services, and have been able to work on a few vCD engagements for VMware as a result, so I felt it was important to have this. Oddly enough assignments I’ve had were not with Cloud Providers, but I’ve gotten to see some pretty unique ways the platform is used outside that space.
I am focusing on prepping for the vSphere VCAP design and deploy certifications. This Cloud Provider specialist cert was a nice distraction. I also have a class scheduled for NSX-T in the fall as I hope to get certified on VCP-NV 2020 as well. I might try and go for the vROps specialist exam this year too, but I really want to get at least one VCAP out of the way first.
I forgot to post here, but in August I finally completed my certification for the VMware Advanced Professional – Data Center Design 2020.
I was hoping that the VCAP- Deploy 2020 would also be made virtual, but I will have to schedule something at a testing facility.
I was able to attend a NSX-T 3.0 Data Center: Install, Configure, Manage course. However the week that I scheduled it, my customer had an outage, so I wasn’t able to focus 100% on this.
I was also able to take an NSX-T 3.0 live fire course, but this too came second to my customer’s needs. Such is Professional Services life I suppose. I was hoping to get the NSX cert this year, but it’s looking like I will need to push that off until 2021.
In my last blog, I walked through the process of Deploying the vCloud Director Appliance v10, and today’s blog will feature the process of deploying two additional standby appliances to create an HA database configuration for vCD. To get an idea of what that architecture would look like, I’ll rip this excellent diagram from VMware’s own documentation.
Deploying additional appliances are pretty straight forward, so lets get started.
1 – Find and upload the OVF for vCD.
2 – Name the VM, select the datacenter and virtual machine folder.
3 – Select the compute cluster
4 – The primary appliance has already been deployed. It is important to note that the same size standby appliance has to be deployed. Because our first primary appliance was deployed as small, so to shall the standby appliance. VMware’s sizing guide and be found here.
5 – Select desired storage disk format and storage where the appliance will reside.
6 – Configure the networks for each network interface, keeping in mind that they will be in reverse order as discussed before.
7 – Fill out the template customization page just like before. Remember all fields including the administrator email are required.
Note: – Be sure to use the same “System name” that was used for the original vCD primary appliance deployment. – For the “Installation ID” section, make sure this value reflects increases with the number of appliance being deployed. In this demonstration I am deploying the 2nd and 3rd appliances, so the installation IDs would be 2 and 3 respectively.
8 – On the summary page, verify the deployment and click finish.
9 – Before starting the appliance, it may be a good idea to take a snapshot. Once the appliance has been started, and the configuration scripts attempt to run and fail, the appliance will need to be redeployed. I’d also take a snapshot of the primary appliance, to roll back any failed attempts to join.
10 – Once you have started the appliance, watch for the “Guest OS Initialization Script”. This should take a couple of minutes to run in order to be successful. If it runs for less than 10 seconds, then there was a problem and the appliance will need to be redeployed.
11 – After the appliance boots, look at the /opt/vmware/var/log/vcd/setupvcd.log to validate a successful cluster join. This log can also be used if the appliance deployment failed.
A successful join would look something like this:
12 – Now deploy the 3rd standby appliance using steps 1 through 11.
13 – Once the 3rd appliance has been deployed, it would be a good idea to log into the primary appliance’s 5480 page to validate the health of the new DB cluster.