Month: December 2021

Tanzu Deployment on vSphere 7 with NSX-T.

CaptainvOPs

Blog Date: December 7, 2021

VMware vCenter Server 7.0 Update 2d used.
VMware NSX-T Data Center 3.1.3.1 used.

Assumptions:

In a previous post titled vSphere with Tanzu on VMware Cloud Foundation/vSphere with NSX-T requirements, I went over the requirements I pass along to customers, along with the supporting VMware documentation, and this post assumes those requirements and those in the VMware documentation have been met. The same networking requirements exist here for standard vSphere 7 deployments with NSX-T.

  1. Validate and deploy an NSX-T edge cluster. For more information see: Configuring NSX-T Data Center for vSphere with Tanzu.
  2. Validate/Add NSX-T Network Segments
  3. Validate/Configure NSX-T IP Prefixes on the Tier-0 Gateway
  4. Validate/Configure NSX-T Route Maps on the Tier-0 Gateway
  5. Validate MTU greater than or equal to 1600 on all networks that will carry Tanzu traffic i.e. management network, NSX Tunnel (Host TEP, Edge TEP) networks, and the external network.
  6. Create a Subscribed Content Library for vSphere with Kubernetes.
  7. Create Storage Policies for vSphere with Tanzu.
  8. Deploy vSphere with Tanzu:

Deployment Steps:

In the vSphere Client, select Menu > Workload Management.

Click Get Started. (The Enable Workload Management wizard opens.)

On the vCenter Server and Network section, select NSX-T. Click Next.

On the Select a Cluster section, select the ESXi cluster to support vSphere with Tanzu.

Next, select the size of the control plane. Click Next.

This image has an empty alt attribute; its file name is vcf-deploy7.png

Fill in the Management Network details.

Scroll down, and fill in the Workload Network details.  As mentioned in a previous post, I will argue that the API Server endpoint FQDN entry is mandatory when applying a certificate. NOTE: The Pod and Service CIDRs are non-routable. The UI provides default values that can be used, otherwise you specify your own. The Ingress and Egress CIDRs will be routable networks defined by the network team.  Click Next.

Select the storage policy for Control Plane Nodes, Ephemeral Disks, Image cache. vSAN Default Storage Policy can be used if only storage/cluster provided. Click Next.

That’s it.  Click Finish.  The Tanzu deployment will now proceed (The entire process can take up to 45 minutes to complete).

The Control Plane Node IP address is the same API Server Endpoint  we referred to earlier in this post. This will be the end point where you can download and install the vSphere plugin and the vSphere docker credential helper. To validate connectivity, simply open a web browser and go to the IP address http://<ip-address&gt;

If you are not able to reach the Control Plane Node IP address/API Server Endpoint, it is possible that you might have invalid MTU settings in your environment that will require further troubleshooting. I did come across this at a customer site, and documented the MTU troubleshooting process here. Good luck.

In my next post, I will cover how to configure your first namespace.

vSphere with Tanzu on VMware Cloud Foundation – Configure VMware Photon OS Developer VM

CaptainvOPs

Blog Date: December 5, 2021 Updated: August 8, 2022
VMware Cloud Foundation 4.3.1 used.
VMware vCenter Server 7.0 Update 2d used.
VMware NSX-T Data Center 3.1.3.1 used.
VMware Photon OS 3.0

On engagements with customers, I’ll have them deploy a developer VM where we can work and I can get them started on their Tanzu and Kubernetes journey. This one VM will have docker, docker credential helper, and the Tanzu Kubernetes CLI installed. For the purpose of this blog series, I’ll do the same.

Getting Started with Photon OS and Installing Docker

The first step was to deploy the Photon OS ova: https://github.com/vmware/photon/wiki. This URL has all of the instructions on getting started as well as running Docker which only requires two commands:

The Docker service needs to be set up to run at startup. To do so, input the following commands:

$ sudo systemctl start docker
$ sudo systemctl enable docker

(Optional) Once that completes, run the following commands to allow docker to run as non-root:

$ sudo groupadd docker
$ sudo usermod -aG docker $USER
$ newgrp docker

The following command will start docker if it is not already running. Likewise you can do a status instead of a start:

$ systemctl start docker

Downloading The Kubernetes CLI

First, if this is going to be a shared box, it will be a good idea to create a directory where we can place the files:

$ mkdir -p /opt/vsphere-plugin

If needed you can locate the control plane node IP address from the workload management section in vSphere.

The Kubernetes CLI can be downloaded from the https:// via wget.

$ wget https://<cluster_ip>/wcp/plugin/linux-amd64/vsphere-plugin.zip

Unzip the vsphere-plugin.zip to the ‘/opt/vsphere-plugin’ directory we created before.

$ unzip vsphere-plugin.zip -d /opt/vsphere-plugin

Configure the environment variable PATH to include the extracted ‘opt/vsphere-plugin’ and set up tab auto completion.

$ echo 'export PATH=/opt/vsphere-plugin:$PATH' >> ~/.bash_profile
$ echo 'source <(kubectl completion bash)' >> ~/.bash_profile

cat the ~/.bash_profile file to verify the new entries. The output should look something like:

$ cat ~/.bash_profile
export PATH=/opt/vsphere-plugin:$PATH source <(kubectl completion bash)

Install and Configure the vSphere Docker Credential Helper

The vSphere docker credential helper helper cli is used to securely push/pull container images to and from the embedded harbor registry. Please see VMware’s official documentation Install the vSphere Docker Credential Helper and Connect to the Registry for more information.

First, if this is going to be a shared box, it will be a good idea to create a directory where we can place the files:

$ mkdir -p /opt/vsphere-docker-credential-helper

From the developer VM, use the kubectl CLI to connect to the vSphere with Tanzu control plane as the authenticated user.

$ kubectl vsphere login --server <cluster_ip> -u <username@example.domain>

To download the vsphere-docker-credential-helper.zip package for Linux operating systems, run the wget command.

$ wget https://<cluster-ip>/wcp/helper/linux-amd64/vsphere-docker-credential-helper.zip

Run the unzip command to extract the downloaded zip package to the custom directory created in a previous step.

$ unzip vsphere-docker-credential-helper.zip -d /opt/vsphere-docker-credential-helper

Now we need to configure the docker client to use the embedded harbor registry cert. Please see VMware’s Document Create Configure a Docker Client with the Embedded Harbor Registry Certificate for more information.

Create a directory path for the private registry in /etc/docker/certs.d/ that corresponds to the IP address of the Harbor instance.

$ mkdir /etc/docker/certs.d/IP-address-of-harbor/

We need to download the certificate for the embedded harbor registry. VMware also has this documented under Download and Install the Embedded Harbor Registry Certificate. For this example I’ll use the vSphere client method.

Select the vCenter cluster where Workload Management and the embedded Harbor Registry are enabled.
– Select Configure > Namespaces > Image Registry.
– In the Root certificate field, click the link Download SSL Root Certificate.
– Save the root-certificate.txt, and rename it to something like ca.crt.

Copy the embedded Harbor Registry ca.crt file that you downloaded to the /etc/docker/certs.d/IP-address-of-harbor/ created in the previous step.

That directory should now look something like:

/etc/docker/certs.d/IP-address-of-harbor/ca.crt

Restart the docker service so that the new certificate is used:

$ systemctl restart docker

To test that the docker credential helper is working, you can log into the embedded harbor registry using your fully qualified domain credentials. As long as you don’t get a certificate trust error, you are good to go.

$ docker-credential-vsphere login <harbor_ip>

This blog should have prepped the Developer VM (Photon OS) that we will be using going forward. There will be a future blog post on pushing a docker image to the embedded harbor registry, but I am not going to cover this here. In my next post, I’ll walk through the steps of installing a Tanzu Kubernetes Cluster inside the namespace we deployed using this VM. Stay tuned.

vSphere with Tanzu on VMware Cloud Foundation – Enable Embedded Harbor Registry

CaptainvOPs

Blog Date: December 2, 2021
VMware Cloud Foundation 4.3.1 used during deployment.

In my last post, I went over the process of Configuring Your First Kubernetes Namespace. In this post I will go over the simple steps to enabling the embedded harbor registry. There are basically two methods to deploying a Harbor registry; the embedded harbor which I will be showing in this post, and Deploying Harbor Registry as a Shared Service.

Note: To use the embedded Harbor Registry, you must deploy the Supervisor Cluster with NSX-T Data Center as the networking solution.

Procedure:

In the vSphere client/workload domain that has Tanzu deployed, select the compute cluster, click on the configure tab, and then in the center menu, scroll down until you find Namespaces. Under Namespaces, select “Image Registry”.

Click Enable Harbor.

In the Select Storage Policies window, select K8S Storage Policy and click OK

The embedded Harbor Registry begins deploying.

The deployment can take up to 20 minutes depending on how large the cluster is, but I have seen the deployment take less than 10 minutes for small clusters of four.

You’ll find the IP address and link to the Harbor UI on this page once the deployment completes. We’ll come back to this in a later post. If you’d like, you can log into the harbor registry UI with the user and or group account that was defined in the namespace permissions section.

In my next post, I’ll go over the steps of getting the Ubuntu VM ready, which I will either refer to as the developer box or developer workstation. We’ll get docker installed, the vsphere plugin which has the Kubernetes cli, and the docker credential helper to start with. Later on we’ll also install some TKG extensions like helm, kapp, and ytt.

vSphere with Tanzu on VMware Cloud Foundation – Configuring Your First Kubernetes Namespace.

CaptainvOPs

Blog Date: December 2, 2021
VMware Cloud Foundation 4.3.1 Used During Deployment.

In my previous post, I described how to deploy vSphere with Tanzu on a VMware Cloud Foundation 4.3.1 instance. In this post I will describe how to configure your first namespace.

Procedure:

Access the vSphere client. Select Menu > Workload Management > Namespaces.

Click Create Namespace.

Expand the inventory tree and select the compute cluster.

As am example, you can enter namespace-01 as your namespace name. (The name must be in a DNS-compliant format (a-z, 0-9, -)).

Click Create. ( The namespace is created and shows a Config Status of Running and a Kubernetes Status of Active.)

Select the Don’t show for future workloads check box.

Click Got It.

Now we can move on to the next section and apply permissions, storage and VM class.

  1. Click Add Permissions
    1. Identity source: <make selection>
    2. User/Group Search: <customer specific>. In this example, I have created a vsphere.local account. You can easily use an active directory account or group here.
    3. Role: <customer specific>. In this example, I have chosen “can edit” that way I can create and destroy things inside the namespace.
    4. Click Ok
    5. (Rinse-wash-repeat as necessary)

Click Add Storage and add the storage policy. 

The namespace is configured with a storage policy and user permissions.  The assigned storage policy is translated to a Kubernetes storage class.

Under VM Service, click Add VM Class. Here we need to associate a VM class with the namespace, that will allow developers to self-service VMs in the namespace. This gives vSphere administrators flexibility with resources available in the cluster. In this example, best-effort-xsmall was chosen because this is a nested lab environment. You should work with your developers to determine the best sizing strategy for the containerized workloads.

Now that the Namespace, Storage, and VM Class policies have all been defined, your window should look something like:

That’s it. Technically we can start deploying workloads to the new namespace. However, because I am already logged into vSphere, I like to enable the embedded Harbor registry. In my next post, I’ll go over the simple process of how to enable the embedded harbor registry.

vSphere with Tanzu on VMware Cloud Foundation – Deployment.

CaptainvOPs

Blog Date: December 1, 2021

VMware Cloud Foundation 4.3.1 Used During Deployment:

Assumptions:

In my previous post vSphere with Tanzu on VMware Cloud Foundation/vSphere with NSX-T Requirements, I went over the requirements I pass along to customers, along with the supporting VMware documentation, and this post assumes those requirements and those in the VMware documentation have been met.

Procedure:

  1. Validate/Create a Storage Policy for vSphere with Tanzu, if default vSAN policy wont be used.
  2. Validate Deployed NSX Edge Cluster.
  3. Validate/Add NSX-T Network Segments
  4. Validate/Configure NSX-T IP Prefixes on the Tier-0 Gateway
  5. Validate/Configure NSX-T Route Maps on the Tier-0 Gateway
  6. Validate MTU greater than or equal to 1600 on all networks that will carry Tanzu traffic i.e. management network, NSX Tunnel (Host TEP, Edge TEP) networks, and the external network.
  7. Create a Subscribed Content Library for vSphere with Kubernetes.
  8. Deploy vSphere with Tanzu:

Deployment Steps:

After you have configured VM (storage) policies in vSphere and added segments in NSX-T Data Center, you can deploy vSphere with Tanzu.  SDDC Manager first validates your environment then redirects you to the vSphere Client where you complete the deployment. From the SDDC manager UI, navigate to Solutions and select Deploy.

Select All to have SDDC manager run a deployment prerequisites check.  Click Begin.

Select the desired cluster for the Tanzu workload.  Click Next. 

The SDDC manager will begin running the Validation. All Statuses should succeed .  Else troubleshoot/Retry.  Click Next.

After successful validation, SDDC will switch you over to the vSphere client to complete the deployment.

In the vSphere client, select the desired cluster to enable Tanzu.  Click Next.

Next, select the size of the control plane. Click Next.

Fill in the Management Network details.

Scroll down, and fill in the Workload Network details.  As mentioned in a previous post, I will argue that the API Server endpoint FQDN entry is mandatory when applying a certificate. NOTE: The Pod and Service CIDRs are non-routable. The UI provides default values that can be used, otherwise you specify your own. The Ingress and Egress CIDRs will be routable networks defined by the network team.  Click Next.

Select the storage policy for Control Plane Nodes, Ephemeral Disks, Image cache. vSAN Default Storage Policy can be used if only storage/cluster provided. Click Next.

That’s it.  Click Finish.  The Tanzu deployment will now proceed (The entire process can take up to 45 minutes to complete).


One the process is complete, we will see a config status of running with a green check. The cluster will have a yellow triangle because we have not assigned a license yet.

The Control Plane Node IP address is the same API Server Endpoint  we referred to earlier in this post. This will be the end point where you can download and install the vSphere plugin and the vSphere docker credential helper. To validate connectivity, simply open a web browser and go to the IP address http://<ip-address&gt;

From here, you can download the CLI plugin for windows.

If you are not able to reach the Control Plane Node IP address/API Server Endpoint, it is possible that you might have invalid MTU settings in your environment that will require further troubleshooting. I did come across this at a customer site, and documented the MTU troubleshooting process here. Good luck.

In my next post, I will cover how to configure your first namespace.

vSphere with Tanzu on VMware Cloud Foundation/vSphere with NSX-T Requirements

CaptainvOPs

Blog Date: December 1, 2021

VMware Cloud Foundation 4.3.1 Used During Deployment:

Summary requirements:

– For VMware Cloud Foundation 4.3.1 software requirements, please refer to the Cloud Foundation Bill of Materials (BOM)
– For vSphere with Tanzu, it is recommended to have at least three hosts with 16 CPU Cores per host (Intel), 128GB per host, 2x 10GBe Nics per host, and a shared datastore of at least 2.5 TB.
– This post assumes VCF 4.3.1 has already been deployed following recommended practices.

System Requirements for Setting Up vSphere with Tanzu with NSX-T Data Center

The following link to VMware’s documentation should be reviewed prior to installation of Tanzu. This link includes networking requirements.
Tanzu with NSX-T Data Center Requirements

Specific Tanzu Call-outs:

IP Network

POD CIDR

Services CIDR

Ingress CIDR

Egress CIDR

Management Network

IPv4 CIDR Prefix

Greater than or equal to /20

Greater than or equal to /22

Greater than or equal to /24

Greater than or equal to /24

5 consecutive IPs

Justification:

POD CIDR: Dedicate a /20 subnet for pod networking. Private IP space behind a NAT that you can use in multiple Supervisor Clusters.Note: when creating TKG clusters, the IP addresses used for the K8s nodes are also allocated from this IP block.
Services CIDR: Dedicate a /22 subnet for services. Private IP space behind a NAT that you can use in multiple Supervisor Clusters.
Ingress CIDR: TKGS sources the ingress CIDR for allocating routable addresses to the Kubernetes clusters’ API VIP, ingress controller VIP, and service-type Load-Balancer VIP(s). Note: This subnet must be routable to the rest of the corporate network.
Egress CIDR: TKGS sources the egress CIDR to enable outbound communication from namespace pods. NSX-T automatically creates a source network address translation (SNAT) entry for each namespace, mapping the pod network to the routable IP address, respectively. Note: This subnet must be routable to the rest of the corporate network
Management Network: Five consecutive IP addresses on the Management network are required to accommodate the Supervisor VMs.
MTU: Greater than or equal to 1600 on all networks that will carry Tanzu traffic i.e. management network, NSX Tunnel (Host TEP, Edge TEP) networks, and the external network.


Note:  Kubernetes cluster requires identifying private IPv4 CIDR blocks for internal pod networks and service IP addresses. The Pods CIDR and Service CIDR blocks cannot overlap with IPs of Workload Management components (vCenter, ESXi hosts, NSX-T components, DNS, and NTP) and other data center IP networks communicating with Kubernetes pods. The minimum Pods CIDR prefix length is /23, and the minimum Service CIDR prefix length is /24.

Also: If using Tanzu Mission Control, TKG cluster components use TCP exclusively (gRPC over HTTP to be specific) to communicate back to Tanzu Mission Control with no specific MTU outbound requirements (TCP supports packet segmentation and reassembly).

Deploy an NSX-T Edge Cluster

The following link to VMware’s documentation should be reviewed, and can be used to deploy the necessary NSX Edge Cluster prior to the Tanzu deployment.
Deploy an NSX-T Edge Cluster

Enterprise Service Requirements

  • DNS: System components require unique resource records and access to domain name servers for forward and reverse resolution.
  • NTP: System management components require access to a stable, common network time source; time skew < 10 seconds.
  • DRS and HA: Need to be enabled in the vSphere cluster
  • (Useful but Optional): Ubuntu developer VM with docker installed for use while interacting with Tanzu.

In my next post, I’ll go over deploying vSphere with Tanzu on VCF with NSX-T.