VMworld 2018 is right around the corner! Where will you be?

June 4, 2018September 9, 2018 CaptainvOPs

It’s almost that time a year again….some might even call it that special time of year where VMware geeks from across the globe converge on VMworld. One might even consider this summer camp, and like any who have experienced this before, you meet new people in the vCommunity, make friends, and part ways after the week of technical sessions, social gatherings, and just the straight up shop talking, war story sharing, and the sharing of ideas. Personally, this will be my third year attending, and I am super excited to be going. This conference means enough to me that, due to other circumstances that happened early this year, I purchased my own pass so to ensure that I wouldn’t miss out.

Now is the perfect time to cash in on those early bird discounts on conference passes, good until June 15. Why wouldn’t you want to save a couple hundred dollars on one of the best IT conferences of the year? For an individual, it’s $1,795 vs $2,095. That’s before other discounts that may be applied like vmug memberships, or the discount for VMware Certified Professionals who hold an active VCP.

So, why go to VMworld?

I think for many first timers, there’s a certain electricity, and excitement about going. Let me be the first to tell you, that feeling…. never really goes away. Like the past couple of years, VMworld in the US will be held once again in Las Vegas.

I personally love coming to VMworld and have looked forward to it every year. There’s always good energy here; the minute you get off the plane, it is happy. Every experience I’ve had here is fun, and people genuinely are in a good mood. This conference gives attendees the chance to attend VMware lead, and partner lead sessions on platforms you may have thought about using or are currently using. These sessions are meant to share best practices with the community, transfer knowledge in ways to use VMware platforms, and also give you a chance to ask the experts, many of whom work for VMware, and in some cases, are very involved with the development of the platforms you use.

VMworld is not just about attending sessions however. This conference gives you the unique opportunity to network with other IT professionals from across the globe and establish relationships that you would otherwise never be able to do. Like it did for me, this conference may also inspire you to join the vCommunity, a thriving community of professionals who not only share their knowledge with others, but who also need help themselves. I think we can all agree that no two environments/businesses are alike, and we have all used VMware’s platforms in ways that were intended, and in ways that even VMware might not have ever considered. Members of the vCommunity take it upon themselves to share their experiences with others, through blogs, social media, and support forums to help others. This conference gives us a chance to get together, share war stories from our time in the trenches, and many times, you will find attendees getting together to engineer and develop something cool.

VMware {code} group has even put together a hackathon, where members from the vCommunity can get together while at VMworld, to develop some amazing things, and sometimes there are even prizes to be had for the coolest of the cool ideas. But don’t let those words “code” or “hackathon” scare you. These sessions are not just for developers! Sure it will certainly help, but the power of the community, enables you to participate in these teams anyways. You may not be able to contribute code, but you can still contribute ideas to the team, and you might even pick up a few coding skills in the fun. Let’s face it; some pretty cool ideas are cooked up during hackathons. VMware’s internal hackathon cooked up the idea to bring VR into the datacenter, and allow you to virtually move your workloads from On-Premises Data Centers, into the cloud. It’s freakin VR man! How cool is that?

The VMworld conference also affords you the opportunity to attend instructor lead labs, along with VMware’s hands on labs that you can also experience from home. While at the conference, there will be many vendors out on the floor where you can experience new products, ask questions about products that you already use, and lets not forget the vendor haul crawl where there will be free adult beverages, snacks, and cool swag vendors are giving out. All can be found in the solutions exchange area.

I’m not going to lie, the parties at VMworld are pretty wild too. Not saying that should be the only reason you go, but it is a good way to mingle with other conference attendees, jam out to some good music, and of course escape the Las Vegas heat. VMworld of course wraps up with it’s own party, before the last day of the conference.

So what are you waiting for? I can’t think of any reason not to attend the US 2018 VMworld in Las Vegas, August 26th – 30th, or the UK 2018 VMworld in Barcelona, November 5th – 8th. Follow this link here, and I will see you at the conference in Las Vegas! Remember to take advantage of those early bird rates, good until June 15th! REGISTER HERE FOR VMWORLD 2018

Enable TLS v1 In vCloud Director 8.20 and vCloud Availability 1.0

December 18, 2017December 18, 2017 CaptainvOPs

VMware’s vCloud Director (vCD) and vCloud Availability (vCAV) only come with TLS v1.1 and 1.2 enabled out of the box. This process will show you how to enable TLS v1. If more information is needed, please visit VMware’s Documentation on vCloud Director 8.20, or the following KB2145796. This work should be completed after hours as you would inevitably be moving VCD proxy service from one cell to another, and this could cause a brief outage for customers. This process will require taking the cell offline, so do each cell one at a time starting with a cell not running the inventory service

Open an SSH session to a VCD cell, or vCAv cloud proxy cell, and su to root
Change to the ‘ /opt/vmware/vcloud-director/bin/ ‘ directory
Use the Cell Management Tool to quiesce the cell. This will move active jobs over to another cell, and cleanly shutdown the cell. You should make note which VCD cell has the proxy service enabled, and avoid that cell until last.

# ./cell-management-tool -u administrator cell --quiesce true

Get the status of any running jobs on each cell. ** Verify Job count = 0 | Is Active = false | In Maintenance Mode = false

# ./cell-management-tool -u administrator cell --status

Example Output:

Job count = 0
Is Active = false In Maintenance Mode = false

Shut the cell down to prevent any other jobs from becoming active on the cell.

# ./cell-management-tool -u administrator cell --shutdown

Example Output:

Cell successfully deactivated and all tasks cleared in preparation for shutdown Stopping vmware-vcd-watchdog:                              [  OK  ] Stopping vmware-vcd-cell:                                  [  OK  ]

Run the following command on the vCD cell in /opt/vmware/vcloud/bin/ to enable TLS1

# ./cell-management-tool ssl-protocols -d SSLv3,SSLv2Hello

Start the cell service, and validate that a vCD cell has the listener service running from the UI, and that vCenter is connected to one of the cells.

# service vmware-vcd start

To validate that TLS v1 has been enabled on the vCD cell, or vCAV cloud proxy cell, run the following command

# ./cell-management-tool ssl-protocols -l

Example output

Allowed SSL protocols:
* TLSv1.2
* TLSv1.1
* TLSv1

If you have additional VCD cells, or vCAV cloud proxy cells, repeat this process one at a time.

Upgrade Existing vRealize Operations Manager Add-on/Solution Paks

October 23, 2017October 23, 2017 CaptainvOPs

The following was recorded using a vRealize Operations Manager (VROps) 6.6 cluster, however older versions of VROps can be upgraded the same way.

Log into the vROps environment, go to the Administration tab, and select solutions in the left column.
Here you can see all of the add-on/solutions paks that I have installed in this environment. To upgrade an existing solution, simply click the green plus button.

Browse for the new pak. In this example I have selected “Reset Default Content” option. As the statement suggests, this can override policies, customized alerts, symptoms etc. that may have been customized by your organization, forcing that work to be re-created. However, I like using this option because I get those new changes, and can adjust my monitoring accordingly. Use at your own discretion

Click ‘upload’

Click ‘Next’
Read and accept the EULA if you so desire
Click ‘Next’

Now the installation process will begin. This shouldn’t take longer than 5 minutes.

vrops54

Click Finish

vrops55

Now the latest version of the Add-on/solutions pak is installed and ready for use. In most cases it will just pick up the config from older versions.

Collecting Java Heap dump from vCloud Director Cells

September 27, 2017December 19, 2017 CaptainvOPs

You just need to generate the java heap dump from one of the cells. What you’ll need to succeed:

JCONSOLE
IP tables disabled on the cell you are connecting to.
Disk space available on the cell to accommodate the dump – I believe these can be between 8 and 10 GB in size
Unless an emergency, do this operation outside of normal business hours as it will be CPU intensive for up to 3 minutes, can impact API call performance, and can potentially cause the VCD cell inventory service to hang.

Step #1: Disable iptables on the cell

ssh to the desired cell and run the following command:

# service iptables stop

Step #2: Connect with jconsole (java console)

domain credentials should work here depending on your environment
connect to port: 8999
connect to desired cell

vcd9

If you get this message “Secure connection failed. Retry Insecurely?” just click the ‘insecure’ button to continue

Step #3: Generate the heap dump

On the MBeans tab, in the com.sun.management/HotSpotDiagnostics object, select the Operation section.
In dumpHeap parameters, enter the following information:
p0: [heap-output-path]

p1: true – do a garbage collection before dump heap

For example:

p0: /opt/vmware/vcloud-director/vcd_cell_name_heap-dump-file.hprof

p1: true
Click the dumpHeap button.

vcd11

There will be no indication that the heapdump completes. I just watch the size of the file until the growth stops on the cell. This process typically takes less than two minutes.

Step #4: Cleanup and send-off

Locate the heap dump in /opt/vmware/vcloud-director/ and move off to a location where you can compress and upload to VMware FTP site as you would for logs.
Start the iptables on the cell: # service iptables start

Upgrading VMware vCloud Director to 8.20

September 25, 2017September 28, 2017 CaptainvOPs

This document was creating while upgrading an existing vCloud Director 8.10.1 environment with an Oracle database, and multiple cloud cells.

After downloading the latest version of vCloud Director 8.20 for service providers, SCP the upgrade to all VCD cells. You can review the release notes here.

What you’ll need to do before getting started:

SSH into each cell and ‘sudo su -‘ to root
move the bin to the root directory
chmod +x vmware-vcloud-director-distribution-8.20.0-5515092.bin
I strongly advise opening an support request with VMWare before proceeding with the upgrade. You may not need it, but it comes in handy having one logged beforehand.

Maintenance – Shutdown the cells

1. Open an SSH session into each VCD cell

2. Sudo to root using the following command:

# sudo su -

3. Change to the vcloud-director/bin/ directory

# cd /opt/vmware/vcloud-director/bin/

4. Use the Cell Management Tool to quiesce the cell. This will move active jobs over to another cell.

# ./cell-management-tool -u administrator cell --quiesce true

5. Get the status of any running jobs on each cell. ** Verify Job count = 0 | Is Active = false | In Maintenance Mode = false

# ./cell-management-tool -u administrator cell --status

Example Output:

vcd6

6. Shut the cell down to prevent any other jobs from becoming active on the cell. This command will also allow active jobs to cleanly finish

# ./cell-management-tool -u administrator cell --shutdown

Example Output:

vcd7

7. Get a status on the cells to be sure everything is down

# service vmware-vcd status

8. Now complete steps 4 – 7 on the remaining cells to cleanly shutdown the vCD service on all cells.

9. Here is where I would shutdown the VCD cell virtual machines, and database to get a clean snapshot while the environment is powered off

10. Once the database virtual machine is fully up, power-on the VCD cell virtual machines.

11. Log back into the vCloud Director environment to verify functionality before the upgrade.

12. SSH to all VCD cell virtual machines and use the following command to stop the service again on each cell. Here there is an assumption made that we are now well within a maintenance window.

# service vmware-vcd stop

Starting The vCloud Director Upgrade

1. Start with the first cell, and run the first half of the upgrade. DO NOT upgrade the database yet.

# ./vmware-vcloud-director-distribution-8.20.0-5515092.bin

Example Output:

vCD1

2. Respond with: y

Example Output:

vcd2

3. Stop. Now you need to run steps one and two on the rest of the vCloud Director Cells, and install the upgrade. Do them one at a time. DO NOT upgrade the database yet.

4. Now that all cells have been upgraded, go back to the first cell and run the database upgrade.

# ./opt/vmware/vcloud-director/bin/upgrade

Example vCD Database upgrade output:

vcd3

5. Respond with: y

vcd4

6. Start the the first cell by responding with ‘y’

vcd8

7. Manually start the VCD service on the remaining cells

# service vmware-vcd start

8. Get the VCD status of all cells by running the following command on each

# service vmware-vcd status

9. Log into the cell, and watch/wait for vCenter to sync with vCD under the Manage & Monitor section → vCenters. This normally takes 30 minutes or so. Once done the status will change from a spinning circle to a green check mark.

10. Run some environment validation tests to be sure everything is working and is proper, and then delete those snapshots taken earlier.

Performing A Database Health Check On vRealize Operation Manager (vROPS) 6.5

July 28, 2017August 1, 2017 CaptainvOPs

In a previous post I showed how you could perform a healthcheck, and possibly resolve database load issues in vROPs versions from 6.3 and older. When VMware released the vROPS 6.5, they changed the way you would access the nodetool utility that is available with cassandra database.

 $VCOPS_BASE/cassandra/apache-cassandra-2.1.8/bin/nodetool --port 9008 status

For the 6.5 release and newer, they added the requirement of using a ‘maintenanceAdmin’ user along with a password file. The new command to check the load status of the activity tables in a vROPS 6.5+ is as follows:

  $VCOPS_BASE/cassandra/apache-cassandra-2.1.8/bin/nodetool -p 9008 --ssl -u maintenanceAdmin --password-file /usr/lib/vmware-vcops/user/conf/jmxremote.password status

Example output would be something similar to this if your cluster is in a healthy state:

If any of the nodes have over 600 MB of load, you should consult with VMware GSS or a TAM on the next steps to take, and how to elevate the load issues.

Next we can check the syncing status of the cluster to determine overall health. The command is as follows:

$VMWARE_PYTHON_BIN /usr/lib/vmware-vcops/tools/vrops-platform-cli/vrops-platform-cli.py getShardStateMappingInfo

Example output:

The “vRealize Ops Shard” refers to the data nodes, and the Master and Master Replica nodes in the main cluster. The available status’ are RUNNING, SYNCING, BALANCING, OUT_OF_BALANCE, and OUT_OF_SYNC.

Out of Balance and Out of Sync should be enough to open an SR and have VMware take a look.

Lastly, we can take a look at the size of the activity table. You can do this by running the following command:

 du -sh /storage/db/vcops/cassandra/data/globalpersistence/activity_tbl-*

Example Output:

If there are two listed here, you should consult with VMware GSS as to which one can safely be removed, as one would be left over from a previous upgrade.

Get VM Tools Version with VMware’s PowerCLI

May 16, 2017May 16, 2017 CaptainvOPs

I had an engineer visit me the other day asking if there was an automated way to get the current version of VMtools running for a set of virtual machines, and in this case, it was for a particular customer running in our vCenter. I said there most certainly was using PowerCLI.

Depending on the size of the environment, the first option here may be sufficient, although it can be an “expensive” query as I’ve noticed it takes longer to return results. Using PowerCLI, you can connect to the desired vCenter and run the following one-liner to get return output on the console. Here I was looking for a specific customer in vCloud Director, so in the vCenter I located the customers folder containing the VMs. Replace the ‘foldername’ inside the asterisks with the desired folder of VMs. This command would also work in a normal vCenter as well.

Get-Folder -name *foldername* | get-vm | get-vmguest | select VMName, ToolsVersion | FT -autosize

Example output:

vmtools01

You can see that this example that folder has a mix of virtual machines running, some not (no ToolsVersion value returned), and has a mix of VMtools versions running.

What if you just wanted a list of all virtual machines in the vCenter, the whole jungle?

 Get-Datacenter -Name "datacentername" | get-vm | get-vmguest | select VMName, ToolsVersion | FT -autosize

In either case, if you want to redirect output to a CSV add the following to the end of the line

 | export-csv -path "\path\to\file\filename.csv" -NoTypeInformation -UseCulture

Example:

Get-Folder -name *foldername* | get-vm | get-vmguest | select VMName, ToolsVersion | export-csv -path "\path\to\file\filename.csv" -NoTypeInformation -UseCulture

Another method/example of getting the tools version, and probably the fastest is using ‘Get-view’. A much longer string of command-lets, but this would be the ideal method for large environments if a quick return of data was needed, lets say for a nightly script that was least impactful to the vCenter.

 Get-Folder -name *foldername* | Get-VM | % { get-view $_.id } | select name, @{Name=“ToolsVersion”; Expression={$_.config.tools.toolsversion}}, @{ Name=“ToolStatus”; Expression={$_.Guest.ToolsVersionStatus}}

Example Output:

vmtools02

If you are after a list of all virtual machines running in the vCenter, a command similar to this can be used:

 Get-VM | % { get-view $_.id } | select name, @{Name=“ToolsVersion”; Expression={$_.config.tools.toolsversion}}, @{ Name=“ToolStatus”; Expression={$_.Guest.ToolsVersionStatus}}

VMware has put together a nice introductory blog on using get-view HERE

Just like last time, if you want to redirect output to a CSV file just take the following on to the end of the line for either method ie specific folder or entire vCenter:

 | export-csv -path "\path\to\file\filename.csv" -NoTypeInformation -UseCulture

Shutdown and Startup Sequence for a vRealize Operations Manager Cluster

March 20, 2017August 18, 2017 CaptainvOPs

You ever hear the phrase “first one in, last one out”? That is the methodology you should use when the need arises to shutdown or startup a vRealize Operations Manager (vROps) cluster. The vROps master should always be the last node to be brought offline in vCenter, and the first node VM to be started in vCenter.

The proper shutdown sequence is as follows:

FIRST: The data nodes
SECOND: The master replica
LAST: The master

The remote collectors can be brought down at any time. When shutting down the cluster, it is important to “bring the cluster offline”. Thing of this as a graceful shutdown of all the services in a controlled manor. You do this from the appliance admin page

1. Log into the admin ui…. https://<vrops-master>/admin/

2. Once logged into the admin UI, click the “Take Offline” button at the top. This will start the graceful shutdown of services running in the cluster. Depending on the cluster size, this can take some time.

3. Once the cluster reads offline, log into the vCenter where the cluster resides and begin shutting down the nodes, starting with the datanodes, master replica, and lastly the master. The remote collectors can be shutdown at any time.

4. When ready, open a VM console to the master VM and power it on. Watch the master power up until it reaches the following splash page example. It may take some time, and SUSE may be running a disk check on the VM. Don’t touch it if it is, just go get a coffee as this may take an hour to complete.

The proper startup sequence is as follows:

FIRST: The master
SECOND: The master replica
LAST: The data nodes, remote collectors

5. Power on the master replica, and again wait for it to fully boot-up to the splash page example above. Then you can power on all remaining data nodes altogether.

6. Log into the admin ui…. https://<vrops-master>/admin/

7. Once logged in, all the nodes should have a status of offline and in a state of Not running before proceeding. If there are nodes with a status of not available, the node has not fully booted up.

8. Once all nodes are in the preferred state, bring the cluster online through the admin UI.

Alternatively…..

If there was a need to shutdown the cluster from the back-end using the same sequence, but you should always use the Admin UI when possible:

Proper shutdown:

FIRST: The data nodes
SECOND: The master replica
LAST: The master

You would need to perform the following command to bring the slice offline. Each node is considered to be a slice. You would do this on each node.

# service vmware-vcops-web stop; service vmware-vcops-watchdog stop; service vmware-vcops stop; service vmware-casa stop

$VMWARE_PYTHON_BIN /usr/lib/vmware-vcopssuite/utilities/sliceConfiguration/bin/vcopsConfigureRoles.py --action=bringSliceOffline --offlineReason=troubleshooting

If there was a need to startup the cluster from the back-end using the same sequence, but you should always use the Admin UI when possible:

Proper startup:

FIRST: The master
SECOND: The master replica
LAST: The data nodes, remote collectors

You would need to perform the following command to bring the slice online. Each node is considered to be a slice. You would do this on each node.

# $VMWARE_PYTHON_BIN $VCOPS_BASE/../vmware-vcopssuite/utilities/sliceConfiguration/bin/vcopsConfigureRoles.py --action bringSliceOnline

# service vmware-vcops-web start; service vmware-vcops-watchdog start; service vmware-vcops start; service vmware-casa start

If there is a need to check the status of the running services on vROps nodes, the following command can be used.

# service vmware-vcops-web status; service vmware-vcops-watchdog status; service vmware-vcops status; service vmware-casa status

Restarting Syslog Service on ESXi

February 20, 2017February 19, 2017 CaptainvOPs

Syslogs, we all use them in some form or another, and most places have their syslogs going to a collection server like Splunk or VMware’s own vRealize Log insight. In the event you have an alert configured that notifies you when an ESXi host has stopped sending syslogs to the logging server, or you get a “General System Error” when attempting to change the syslog.global.logdir configuration option on the ESXi host itself, you should open a secure shell to the ESXi server and investigate further.

1. Once a secure shell has been established with the ESXi host, check the config of the vmsyslogd service, and that the process is running by using the following command:

# esxcli system syslog config get

If the process is running and configured, output received would be something similar to:

Default Network Retry Timeout: 180
Local Log Output: /vmfs/volumes/559dae9e-675318ea-b724-901b0e223e18/logs
Local Log Output Is Configured: true
Local Log Output Is Persistent: true
Local Logging Default Rotation Size: 1024
Local Logging Default Rotations: 8
Log To Unique Subdirectory: true
Remote Host: udp://logging-server.mydomain-int.net:514

2. If the process is up, look for the current syslog process with the following command:

# ps -Cc | grep vmsyslogd

3. If the service is running, the output received would be similar to the example below. If there is no output, then the vmsyslogd service is dead and needs to be started. Skip ahead to step 5 if this is the case.

132798531 132798531 vmsyslogd            /bin/python -OO /usr/lib/vmware/vmsyslog/bin/vmsyslogd.pyo
132798530 132798530 wdog-132798531       /bin/python -OO /usr/lib/vmware/vmsyslog/bin/vmsyslogd.pyo

4. In this example, we would need to kill the vmsyslogd and wdog processes before we can restart the syslog daemon on the host.

# kill -9 132798530
# kill -9 132798531

5. To start the process issue the following command:

# /usr/lib/vmware/vmsyslog/bin/vmsyslogd

6. Verify that the process is correctly configured and running again.

# esxcli system syslog config get

Default Network Retry Timeout: 180
Local Log Output: /vmfs/volumes/559dae9e-675318ea-b724-901b0e223e18/logs
Local Log Output Is Configured: true
Local Log Output Is Persistent: true
Local Logging Default Rotation Size: 1024
Local Logging Default Rotations: 8
Log To Unique Subdirectory: true
Remote Host: udp://logging-server.mydomain-int.net:514

7. Log into the syslog collection server and verify the ESXi host is now properly sending logs.