[{"content":"","date":"5 May 2026","externalUrl":null,"permalink":"/tags/external-secrets/","section":"Tags","summary":"","title":"External-Secrets","type":"tags"},{"content":"","date":"5 May 2026","externalUrl":null,"permalink":"/tags/k8s/","section":"Tags","summary":"","title":"K8s","type":"tags"},{"content":"","date":"5 May 2026","externalUrl":null,"permalink":"/tags/kubernetes/","section":"Tags","summary":"","title":"Kubernetes","type":"tags"},{"content":"","date":"5 May 2026","externalUrl":null,"permalink":"/tags/openbao/","section":"Tags","summary":"","title":"Openbao","type":"tags"},{"content":"","date":"5 May 2026","externalUrl":null,"permalink":"/posts/","section":"Posts","summary":"","title":"Posts","type":"posts"},{"content":"If you haven\u0026rsquo;t heard of the VCF Secret Store Service I highly recommend taking a look. This is a supervisor service that behind the scenes deploys OpenBao. however it\u0026rsquo;s not just deploying OpenBao, it provides deep integration with VKS, VMs and vSphere Pods. I am not going to outline every feature here, but it\u0026rsquo;s specifically built to make it easier to store and retrieve secrets in a VCF environment securely and seamlessly. This post is going to be dedicated to walking through using the Secret Store with External Secrets Operator(ESO).\nAs a pre-requisite to this you need to deploy the service, the docs here can help out with that. For these specific features you will want to deploy version 9.1.0+25367485 , which you can find linked on the docs or directly here. Once the service is running it will automatically handle a lot of the heavy lifting for us. Specifically the there will be two features that we take advantage of here.\nKeyValueSecret API - when creating a secret using the Secret Store\u0026rsquo;s api(KeyValueSecret) we can have it automatically place the secret into a OpenBao key value mount that will be cluster scoped or supervisor namespace scoped. The secret store service handles the roles and policies required to do this so a user only needs access to the KeyValueSecret api in a namespace. It\u0026rsquo;s also important to know that this API does not store the secret in ETCD, this is passed through directly to OpenBao.\nAutomated k8s auth - the secret store service will automatically create a k8s auth mount for each cluster created, it will also set up a role and policy that allows service accounts in that VKS cluster to access the cluster scoped KV path.\nBetween these two features it will reduce the toil for a developer or platform engineer when accessing secrets in their clusters. Typically setting this up without Secret Store would require direct access to the OpenBao API in order to write secrets along with roles etc. that were setup by the admin to allow users this access. Then for access from the cluster this would require an OpenBao admin to create a k8s auth mount point in OpenBao for every cluster with roles, policy, and have access to the k8s cluster details in order to properly setup that auth endpoint. By automating this process with Secret Store it provides immediate access for users in a supervisor namespace to create secrets without having to have roles manually created and completely removes the manual work that an OpenBao admin would need to do in order to setup k8s auth on the clusters.\nThroughout this post I will be showing what the results look like using the bao cli this is not a requirement for using the Secret Store Service, this is just for educational purposes.\nSetup # Pre-reqs # The Secret Store Supervisor Service Deployed\nA Supervisor namespace\nA VKS cluster created in the namespace\nCreating a secret # In this example we will create a simple secret in the namespace e2e-ns-4cwqh this is going to be a cluster scoped secret. To create a cluster scoped secret simply prefix the secret name with the cluster name. In this case the cluster name is e2e-cls01 , so we will create a secret called e2e-cls01-db-cred . The policy and role that are automatically created when the cluster is created will be set only only allow reads from secrets with this cluster naming convention(more on this later).\nAdd the following yaml to a file called keyvaluesecret.yaml and apply it into the namespace\nkind: KeyValueSecret apiVersion: secretstore.vmware.com/v1alpha1 metadata: name: e2e-cls01-db-cred spec: name: e2e-cls01-db-cred data: - key: username value: admin - key: password value: secretadminpass kubectl apply -f keyvaluesecret.yaml -n e2e-ns-4cwqh We can see the secret was created from both the namespace view and the underlying OpenBao view. Notice that the secret store API does not return the secret values by design and for security reasons.\nkubectl output:\nand the bao CLI output:\nAt this point our secret is created and cluster scoped. Let\u0026rsquo;s take a look at the policies that were auto created for the cluster and namespace.\nReviewing the Policies # As mentioned before one of the big benefits of the Secret Store Service is that it automatically handles auth, roles, and policies for namespaces and clusters. let\u0026rsquo;s use the bao cli to take a look at what has been created for the cluster scoped access. This step is not necessary when using the service, it\u0026rsquo;s purely educational\nHere is the k8s auth endpoint that was automatically generated for our cluster.\nWe can then see what role was associated with this new auth endpoint.\nNow let\u0026rsquo;s look at the role and see what Policy it\u0026rsquo;s associated with. This also shows what K8s service accounts can be used with this in the VKS cluster.\nAnd now we can look at the policy and see what it allows. This shows that it has read capability on the namespace path for anything starting with the cluster name.\nAfter looking at all of this it validates that Secret Store Service created everything we need to be able to access the cluster scoped secrets in our secret store.\nDeploy External Secret Operator # As of writing this there is not an official ESO package for VKS, but that\u0026rsquo;s ok we can use the ESO helm chart without any issues because of course VKS clusters are conformant K8s clusters.\nPrior to deploying let\u0026rsquo;s collect some info, we need to get our Secret Store Service endpoint, this can be found by running the below command in the supervisor namespace context. You can see in the output that we have an external IP, that is what we are going to connect to on https and port 8200.\nWe also will want the CA certificate for the service, we will need it in base64 format so no need to decode it. You can get it with this command. Save both of these details for later.\nkubectl get secret server-cert -n svc-secret-store-domain-c10 -o jsonpath=\u0026#39;{.data.ca\\.crt}\u0026#39; Now to deploy ESO, connect to your VKS cluster context and deploy ESO using the provided helm instructions. The only addition we will make is to add a host alias for the secret store service, this way we can easily trust the cert later on. Use this for the values file:\nglobal: hostAliases: - ip: \u0026#34;10.1.0.8\u0026#34; # replace this with your external IP hostnames: - \u0026#34;secret-store\u0026#34; Create the service account # In order to connect back to the secret store service we need a service account in the VKS cluster. This service account will also need a specific cluster role for \u0026ldquo;auth-delegator\u0026rdquo; create a file called sa.yml and paste the below yaml into it. This will handle creating the service account and the role binding that is needed.\n# service account apiVersion: v1 kind: ServiceAccount metadata: name: secret-store-access namespace: external-secrets --- # Cluster role binding apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: name: ss-auth-binding roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: system:auth-delegator subjects: - kind: ServiceAccount name: secret-store-access namespace: external-secrets Now apply this into the VKS cluster.\n$ kubectl apply -f sa.yaml At this point the service account in the cluster should have access to the cluster scoped secrets in the Secret Store. The last steps are to create the ESO components to fetch the secret.\nCreate the ESO resources # The ClusterSecretStore is a cluster level resource that is going to connect ESO to our Secret Store Service OpenBao instance. This is where the service account comes in. Create a file called css.yaml and put the below yaml into it. Even if the user creating this does not have access to directly look up roles and auth, which they likely won\u0026rsquo;t, these are just templated names that can be derived easily. The mountPath is always kubernetes-\u0026lt;supervisor_ns\u0026gt;-\u0026lt;cluster_name\u0026gt; and the role is always \u0026lt;supervisor_ns\u0026gt;-\u0026lt;cluster_name\u0026gt; .\napiVersion: external-secrets.io/v1 kind: ClusterSecretStore metadata: name: vcf-cluster-store spec: provider: vault: server: \u0026#34;https://secret-store:8200\u0026#34; caBundle: \u0026#34;\u0026lt;base64 bundle from above\u0026gt;\u0026#34; # update this will the bundle from the previous step path: \u0026#34;secret\u0026#34; version: \u0026#34;v2\u0026#34; auth: kubernetes: # This is the path where the cluster auth is mounted mountPath: \u0026#34;kubernetes-e2e-ns-4cwqh-e2e-cls01\u0026#34; # This is the role name we found earlier role: \u0026#34;e2e-ns-4cwqh-e2e-cls01\u0026#34; # The ServiceAccount ESO uses to authenticate serviceAccountRef: name: \u0026#34;secret-store-access\u0026#34; namespace: \u0026#34;external-secrets\u0026#34; $ kubectl apply -f css.yaml After applying this you should see that ESO was able to connect to Secret Store successfully.\nThe Final step is to pull the secret using an externalSecret resource. Create a file called es.yaml with the below content and apply it into the VKS cluster.\napiVersion: external-secrets.io/v1 kind: ExternalSecret metadata: name: db-creds-sync namespace: default spec: refreshInterval: \u0026#34;1h\u0026#34; secretStoreRef: name: vcf-cluster-secretstore kind: ClusterSecretStore target: name: db-cred data: - secretKey: username # The key inside the K8s secret remoteRef: key: e2e-ns-4cwqh/e2e-cls01-db-cred # Path in SecretStore property: username # The key inside SecretStore - secretKey: password remoteRef: key: e2e-ns-4cwqh/e2e-cls01-db-cred property: password After creating this resource we can check to see if we have access to the secret contents in the cluster which should be in the secret called db-cred in the default namespace.\nThis shows that we successfully retrieved the secret from the secret store using k8s auth and ESO.\n","date":"5 May 2026","externalUrl":null,"permalink":"/posts/vcf-secret-store-eso/","section":"Posts","summary":"how to configure ESO on VKS with VCF Secret Store","title":"Secure Kubernetes Secrets on VCF with External Secrets Operator and Secret Store Service","type":"posts"},{"content":"","date":"5 May 2026","externalUrl":null,"permalink":"/tags/","section":"Tags","summary":"","title":"Tags","type":"tags"},{"content":"","date":"5 May 2026","externalUrl":null,"permalink":"/tags/vault/","section":"Tags","summary":"","title":"Vault","type":"tags"},{"content":"","date":"5 May 2026","externalUrl":null,"permalink":"/tags/vcf9/","section":"Tags","summary":"","title":"Vcf9","type":"tags"},{"content":"","date":"5 May 2026","externalUrl":null,"permalink":"/tags/vmware/","section":"Tags","summary":"","title":"Vmware","type":"tags"},{"content":"","date":"5 May 2026","externalUrl":null,"permalink":"/","section":"Will Arroyo","summary":"","title":"Will Arroyo","type":"page"},{"content":"","date":"16 March 2026","externalUrl":null,"permalink":"/tags/argocd/","section":"Tags","summary":"","title":"Argocd","type":"tags"},{"content":"With support for ArgoCD as a service added to VCF I have been using it a lot more for automating my K8s environments. I also heavily use VCF Automation(VCFA) as my main cloud console for my VCF private cloud. I recently came across a feature in VCFA that is really useful, this is the ability to create \u0026ldquo;relying parties\u0026rdquo; this adds the ability to use VCFA and it\u0026rsquo;s backing OIDC provider as an OIDC provider to other applications.\nWith these two features it immediately became clear that there is a good pairing here. I can have ArgoCD OIDC use VCFA as the provider making it so that users can seamlessly login into ArgoCD and as an administrator I can use common roles/groups for access. The rest of this post walks through setting up this integration.\nYour browser cannot play this video. Download video.\nArchitecture # Implementation # Setup OIDC for your Org # The first step in this process is making sure you have OIDC setup for your tenant Org. VCFA has the ability to do per tenant(Org) identity provider configuration. You can configure it with OIDC, SAML, or AD, once this is configured then logging into the tenant goes through your provider of choice and we have access to all of the claims, groups etc. that you setup in your upstream IDP. Ultimately we will use these groups in ArgoCD as well. I am not going to go into detail on how to set up the tenant OIDC in this post, but below is a screenshot of my configuration for Okta integration and here are the official docs to configure it.\nSetup the OIDC Service # This is the \u0026ldquo;relying party\u0026rdquo; I mentioned in the intro. It is basically a way to create OIDC clients that use VCFA as the provider. So in this case because Okta is my backing IDP for VCFA that means the relying party I create will also be able to use Okta for auth, but with far less configuration. Since it all goes through VCFA it will be treated as a single sign on as well so once I log into VCFA then I can seamlessly login to ArgoCD.\nLogin to the the VCFA provider portal as an administrator and go to OIDC services-\u0026gt; Relying Parties Create a new relying party using the DNS name or IP address of the ArgoCD instance that will be deployed. Once you hit save it will generate a client secret, be sure to save that.\nDeploy ArgoCD # In this step, ArgoCD will be deployed as a service and using the new OIDC client it will be integrated into VCFA authentication.\nThe below yaml can be used to deploy ArgoCD and also integrate OIDC with VCFA. Update the fields that have ##UPDATE THIS next to them. This should use details from the previous steps. apiVersion: argocd-service.vsphere.vmware.com/v1alpha1 kind: ArgoCD metadata: name: argocd-dev namespace: infra-ty3qk ##UPDATE THIS spec: applicationSet: enabled: true enableLoadBalancer: true oidc: clientID: 4060b628-c297-49d3-ae0f-31cdcfb9ce86 ##UPDATE THIS clientSecret: mmSKfx8UZN6oYa31hGu95N+t+y1rbEih ##UPDATE THIS enabled: true insecure: true issuer: https://vcf-a.vcf.lab/oidc ##UPDATE THIS name: vcfa requestedIDTokenClaims: groups: essential: true preferred_username: essential: true rbac: policy: | g, \u0026#34;Organization Administrator\u0026#34;, role:admin policyMatchMode: glob scopes: \u0026#39;[groups,roles]\u0026#39; serverSideDiff: true url: https://argocd-dev.vcf.lab ##UPDATE THIS version: 3.0.19+vmware.1-vks.1 Apply the yaml into a supervisor namespace. You should see the ArgoCD pods come up and become healthy.\nAdd DNS. In this example I used argocd-dev.vcf.lab as my DNS auth callback in the relying party config, so ArgoCD needs to be available at that address. You can get the IP address for the ArgoCD Server by doing a kubectl get svc -n \u0026lt;supervisor-ns\u0026gt; and getting the external IP for the server.\nThere are a couple of things to note from the above YAML:\nissuer: https://vcf-a.vcf.lab/oidc - this is your VCFA instance\ng, \u0026quot;Organization Administrator\u0026quot;, role:admin - this is what maps the role from VCFA to the ArgoCD admin role. You can make as many policy rules as you would like and also utilize groups from your upstream IDP. For example I also have a group called argocd-admin in my Okta. I could also add the policy g, \u0026quot;argocd-admin\u0026quot;, role:admin\nscopes: '[groups,roles]' - this tells ArgoCD to get both the groups and the roles from the token and use them for policy mapping.\nurl: https://argocd-dev.vcf.lab - this is a required setting, make sure this matches the callback url without the auth/callback\nValidation # Now that the integration is complete we can test the login and make sure that groups and roles are propagating correctly.\nGo to your ArgoCD server, in my case https://argocd-dev.vcf.lab . You will now see a \u0026ldquo;login with OIDC\u0026rdquo; button. If you are already logged into VCFA it will log you in as this user so be sure to logout if you want to use a different user.\nWhen it redirects to VCFA choose your org that you have setup the OIDC on\nClick \u0026ldquo;Login with OIDC\u0026rdquo; and it will take you through the normal process and redirect you back to ArgoCD.\nValidate your groups and username in ArgoCD. Go to User Info on the left side panel. This is what I see when logging in:\n","date":"16 March 2026","externalUrl":null,"permalink":"/posts/vcfa-argocd-oidc/","section":"Posts","summary":"integrating ArgoCD OIDC auth with VCF Automation","title":"Integrating ArgoCD  authentication with VCF Automation","type":"posts"},{"content":"","date":"16 March 2026","externalUrl":null,"permalink":"/tags/oidc/","section":"Tags","summary":"","title":"Oidc","type":"tags"},{"content":"","date":"16 March 2026","externalUrl":null,"permalink":"/tags/vcf/","section":"Tags","summary":"","title":"Vcf","type":"tags"},{"content":"","date":"27 January 2026","externalUrl":null,"permalink":"/tags/broadcom/","section":"Tags","summary":"","title":"Broadcom","type":"tags"},{"content":"There have been numerous how-to guides over the past few years on building and deploying Windows clusters on the different Kubernetes distributions supported by VMware. This post aims to solidify the understanding of the current most up to date process of doing this on vSphere Kubernetes Service which is the definitive Kubernetes service for VMware. At the time of writing the latest vSphere Kubernetes Release(VKR) is v1.34.1---vmware.1-vkr.4 so that is what this will be based on.\n💡 Note: this is based on VKS 3.5 using VKR 1.34.1. Be sure to check your versions and select the right image builder release based on your environment. Building the Image # Before deploying a cluster, we need to build a windows image. There is a process for this using the VKS image builder , this is what we will use in the next few steps to build the image.\nPre-requisites # Download the Windows ISO for windows server 2022. You can download the evaluation if needed.\nDownload the Windows VMware tools ISO\nUpload the ISOs to a directory on a datastore in your vSphere environment that you will be running this build against. The below script can be used with GOVC to upload the ISOs.\n#!/bin/bash set -e # Exit immediately if a command exits with a non-zero status print_usage() { echo \u0026#34;Usage: $0 \u0026lt;LOCAL_ISO_PATH\u0026gt; \u0026lt;DATASTORE_NAME\u0026gt; \u0026lt;REMOTE_FOLDER\u0026gt;\u0026#34; echo \u0026#34;\u0026#34; echo \u0026#34;Arguments:\u0026#34; echo \u0026#34; LOCAL_ISO_PATH Path to the ISO file on your local machine.\u0026#34; echo \u0026#34; DATASTORE_NAME Name of the vSphere Datastore (e.g., vsanDatastore).\u0026#34; echo \u0026#34; REMOTE_FOLDER Folder path inside the datastore (e.g., ISOs/Linux).\u0026#34; echo \u0026#34;\u0026#34; echo \u0026#34;Example:\u0026#34; echo \u0026#34; $0 ./ubuntu.iso vsanDatastore ISOs/Ubuntu\u0026#34; } check_govc() { if ! command -v govc \u0026amp;\u0026gt; /dev/null; then echo \u0026#34;Error: \u0026#39;govc\u0026#39; is not installed or not in your PATH.\u0026#34; echo \u0026#34;Please install it from: https://github.com/vmware/govmomi/tree/master/govc\u0026#34; exit 1 fi } if [ \u0026#34;$#\u0026#34; -ne 3 ]; then print_usage exit 1 fi LOCAL_ISO=\u0026#34;$1\u0026#34; DATASTORE=\u0026#34;$2\u0026#34; REMOTE_FOLDER=\u0026#34;$3\u0026#34; FILENAME=$(basename \u0026#34;$LOCAL_ISO\u0026#34;) REMOTE_PATH=\u0026#34;$REMOTE_FOLDER/$FILENAME\u0026#34; check_govc if [ ! -f \u0026#34;$LOCAL_ISO\u0026#34; ]; then echo \u0026#34;Error: Local file \u0026#39;$LOCAL_ISO\u0026#39; not found.\u0026#34; exit 1 fi if [ -z \u0026#34;$GOVC_URL\u0026#34; ]; then echo \u0026#34;Error: GOVC_URL environment variable is not set.\u0026#34; echo \u0026#34;Please export GOVC_URL, GOVC_USERNAME, and GOVC_PASSWORD.\u0026#34; exit 1 fi echo \u0026#34;--- Starting Upload Process ---\u0026#34; echo \u0026#34;File: $FILENAME\u0026#34; echo \u0026#34;Datastore: $DATASTORE\u0026#34; echo \u0026#34;Folder: $REMOTE_FOLDER\u0026#34; if govc datastore.ls -ds=\u0026#34;$DATASTORE\u0026#34; \u0026#34;$REMOTE_FOLDER\u0026#34; \u0026amp;\u0026gt; /dev/null; then echo \u0026#34;[OK] Remote folder \u0026#39;$REMOTE_FOLDER\u0026#39; exists.\u0026#34; else echo \u0026#34;[INFO] Remote folder \u0026#39;$REMOTE_FOLDER\u0026#39; does not exist. Creating it...\u0026#34; if govc datastore.mkdir -ds=\u0026#34;$DATASTORE\u0026#34; \u0026#34;$REMOTE_FOLDER\u0026#34;; then echo \u0026#34;[OK] Folder created successfully.\u0026#34; else echo \u0026#34;Error: Failed to create folder \u0026#39;$REMOTE_FOLDER\u0026#39; on datastore \u0026#39;$DATASTORE\u0026#39;.\u0026#34; exit 1 fi fi if govc datastore.ls -ds=\u0026#34;$DATASTORE\u0026#34; \u0026#34;$REMOTE_PATH\u0026#34; \u0026amp;\u0026gt; /dev/null; then echo \u0026#34;Warning: File \u0026#39;$REMOTE_PATH\u0026#39; already exists on datastore \u0026#39;$DATASTORE\u0026#39;.\u0026#34; read -p \u0026#34;Do you want to overwrite it? (y/N): \u0026#34; -n 1 -r echo if [[ ! $REPLY =~ ^[Yy]$ ]]; then echo \u0026#34;[INFO] Upload skipped by user.\u0026#34; exit 0 fi echo \u0026#34;[INFO] Overwriting existing file...\u0026#34; fi echo \u0026#34;[INFO] Uploading \u0026#39;$LOCAL_ISO\u0026#39;... (This may take a while)\u0026#34; if govc datastore.upload -ds=\u0026#34;$DATASTORE\u0026#34; \u0026#34;$LOCAL_ISO\u0026#34; \u0026#34;$REMOTE_PATH\u0026#34;; then echo \u0026#34;\u0026#34; echo \u0026#34;Success: Upload complete!\u0026#34; echo \u0026#34;Location: [$DATASTORE] $REMOTE_PATH\u0026#34; else echo \u0026#34;\u0026#34; echo \u0026#34;Error: Upload failed.\u0026#34; exit 1 fi Example usage:\n./uploadiso.sh ./VMware-tools-windows-12.5.0-23800621.iso cls-wld9-vsan01 isos Setting up the repo # Clone the vks-image-builder repo git clone https://github.com/vmware/vks-image-builder.git cd vks-image-builder Create the windows answers file, the Upstream file can be found here. I have added one below with a few updates. The updates have been marked with comments to highlight them. You will need to also update the areas that have an “Updates this” marker, this is really just updating the Windows password. If you are using the eval version you need to remove the product key from the file. 💡 Note: In the below file there is an addition that I made that adds rules to the windows firewall. I saw an issue where the windows firewall believed I was coming from a public network. This is likely due to my specific network setup so you can leave it there or test without it. It is marked with a comment below. \u0026lt;unattend xmlns=\u0026#34;urn:schemas-microsoft-com:unattend\u0026#34; xmlns:wcm=\u0026#34;http://schemas.microsoft.com/WMIConfig/2002/State\u0026#34;\u0026gt; \u0026lt;settings pass=\u0026#34;windowsPE\u0026#34;\u0026gt; \u0026lt;component name=\u0026#34;Microsoft-Windows-PnpCustomizationsWinPE\u0026#34; processorArchitecture=\u0026#34;amd64\u0026#34; publicKeyToken=\u0026#34;31bf3856ad364e35\u0026#34; language=\u0026#34;neutral\u0026#34; versionScope=\u0026#34;nonSxS\u0026#34;\u0026gt; \u0026lt;DriverPaths\u0026gt; \u0026lt;PathAndCredentials wcm:action=\u0026#34;add\u0026#34; wcm:keyValue=\u0026#34;A\u0026#34;\u0026gt; \u0026lt;Path\u0026gt;a:\\\u0026lt;/Path\u0026gt; \u0026lt;/PathAndCredentials\u0026gt; \u0026lt;/DriverPaths\u0026gt; \u0026lt;/component\u0026gt; \u0026lt;component name=\u0026#34;Microsoft-Windows-Setup\u0026#34; processorArchitecture=\u0026#34;amd64\u0026#34; publicKeyToken=\u0026#34;31bf3856ad364e35\u0026#34; language=\u0026#34;neutral\u0026#34; versionScope=\u0026#34;nonSxS\u0026#34;\u0026gt; \u0026lt;DiskConfiguration\u0026gt; \u0026lt;Disk wcm:action=\u0026#34;add\u0026#34;\u0026gt; \u0026lt;CreatePartitions\u0026gt; \u0026lt;CreatePartition wcm:action=\u0026#34;add\u0026#34;\u0026gt; \u0026lt;Order\u0026gt;1\u0026lt;/Order\u0026gt; \u0026lt;Type\u0026gt;EFI\u0026lt;/Type\u0026gt; \u0026lt;Size\u0026gt;100\u0026lt;/Size\u0026gt; \u0026lt;/CreatePartition\u0026gt; \u0026lt;CreatePartition wcm:action=\u0026#34;add\u0026#34;\u0026gt; \u0026lt;Order\u0026gt;2\u0026lt;/Order\u0026gt; \u0026lt;Type\u0026gt;MSR\u0026lt;/Type\u0026gt; \u0026lt;Size\u0026gt;16\u0026lt;/Size\u0026gt; \u0026lt;/CreatePartition\u0026gt; \u0026lt;CreatePartition wcm:action=\u0026#34;add\u0026#34;\u0026gt; \u0026lt;Order\u0026gt;3\u0026lt;/Order\u0026gt; \u0026lt;Type\u0026gt;Primary\u0026lt;/Type\u0026gt; \u0026lt;Extend\u0026gt;true\u0026lt;/Extend\u0026gt; \u0026lt;/CreatePartition\u0026gt; \u0026lt;/CreatePartitions\u0026gt; \u0026lt;ModifyPartitions\u0026gt; \u0026lt;ModifyPartition wcm:action=\u0026#34;add\u0026#34;\u0026gt; \u0026lt;Order\u0026gt;1\u0026lt;/Order\u0026gt; \u0026lt;Format\u0026gt;FAT32\u0026lt;/Format\u0026gt; \u0026lt;Label\u0026gt;System\u0026lt;/Label\u0026gt; \u0026lt;PartitionID\u0026gt;1\u0026lt;/PartitionID\u0026gt; \u0026lt;/ModifyPartition\u0026gt; \u0026lt;ModifyPartition wcm:action=\u0026#34;add\u0026#34;\u0026gt; \u0026lt;Order\u0026gt;2\u0026lt;/Order\u0026gt; \u0026lt;PartitionID\u0026gt;2\u0026lt;/PartitionID\u0026gt; \u0026lt;/ModifyPartition\u0026gt; \u0026lt;ModifyPartition wcm:action=\u0026#34;add\u0026#34;\u0026gt; \u0026lt;Order\u0026gt;3\u0026lt;/Order\u0026gt; \u0026lt;Format\u0026gt;NTFS\u0026lt;/Format\u0026gt; \u0026lt;Label\u0026gt;Windows\u0026lt;/Label\u0026gt; \u0026lt;Letter\u0026gt;C\u0026lt;/Letter\u0026gt; \u0026lt;PartitionID\u0026gt;3\u0026lt;/PartitionID\u0026gt; \u0026lt;/ModifyPartition\u0026gt; \u0026lt;/ModifyPartitions\u0026gt; \u0026lt;WillWipeDisk\u0026gt;true\u0026lt;/WillWipeDisk\u0026gt; \u0026lt;DiskID\u0026gt;0\u0026lt;/DiskID\u0026gt; \u0026lt;/Disk\u0026gt; \u0026lt;/DiskConfiguration\u0026gt; \u0026lt;ImageInstall\u0026gt; \u0026lt;OSImage\u0026gt; \u0026lt;InstallTo\u0026gt; \u0026lt;DiskID\u0026gt;0\u0026lt;/DiskID\u0026gt; \u0026lt;PartitionID\u0026gt;3\u0026lt;/PartitionID\u0026gt; \u0026lt;/InstallTo\u0026gt; \u0026lt;InstallFrom\u0026gt; \u0026lt;MetaData wcm:action=\u0026#34;add\u0026#34;\u0026gt; \u0026lt;Key\u0026gt;/IMAGE/NAME\u0026lt;/Key\u0026gt; \u0026lt;Value\u0026gt;Windows Server 2022 SERVERSTANDARDCORE\u0026lt;/Value\u0026gt; \u0026lt;/MetaData\u0026gt; \u0026lt;/InstallFrom\u0026gt; \u0026lt;/OSImage\u0026gt; \u0026lt;/ImageInstall\u0026gt; \u0026lt;UserData\u0026gt; \u0026lt;AcceptEula\u0026gt;true\u0026lt;/AcceptEula\u0026gt; \u0026lt;FullName\u0026gt;Administrator\u0026lt;/FullName\u0026gt; \u0026lt;Organization\u0026gt;Organization\u0026lt;/Organization\u0026gt; \u0026lt;ProductKey\u0026gt; \u0026lt;Key\u0026gt;VDYBN-27WPP-V4HQT-9VMD4-VMK7H\u0026lt;/Key\u0026gt; \u0026lt;WillShowUI\u0026gt;OnError\u0026lt;/WillShowUI\u0026gt; \u0026lt;/ProductKey\u0026gt; \u0026lt;/UserData\u0026gt; \u0026lt;EnableFirewall\u0026gt;true\u0026lt;/EnableFirewall\u0026gt; \u0026lt;/component\u0026gt; \u0026lt;component name=\u0026#34;Microsoft-Windows-International-Core-WinPE\u0026#34; processorArchitecture=\u0026#34;amd64\u0026#34; publicKeyToken=\u0026#34;31bf3856ad364e35\u0026#34; language=\u0026#34;neutral\u0026#34; versionScope=\u0026#34;nonSxS\u0026#34;\u0026gt; \u0026lt;SetupUILanguage\u0026gt; \u0026lt;UILanguage\u0026gt;en-US\u0026lt;/UILanguage\u0026gt; \u0026lt;/SetupUILanguage\u0026gt; \u0026lt;InputLocale\u0026gt;0409:00000409\u0026lt;/InputLocale\u0026gt; \u0026lt;SystemLocale\u0026gt;en-US\u0026lt;/SystemLocale\u0026gt; \u0026lt;UILanguage\u0026gt;en-US\u0026lt;/UILanguage\u0026gt; \u0026lt;UILanguageFallback\u0026gt;en-US\u0026lt;/UILanguageFallback\u0026gt; \u0026lt;UserLocale\u0026gt;en-US\u0026lt;/UserLocale\u0026gt; \u0026lt;/component\u0026gt; \u0026lt;/settings\u0026gt; \u0026lt;settings pass=\u0026#34;offlineServicing\u0026#34;\u0026gt; \u0026lt;component name=\u0026#34;Microsoft-Windows-LUA-Settings\u0026#34; processorArchitecture=\u0026#34;amd64\u0026#34; publicKeyToken=\u0026#34;31bf3856ad364e35\u0026#34; language=\u0026#34;neutral\u0026#34; versionScope=\u0026#34;nonSxS\u0026#34;\u0026gt; \u0026lt;EnableLUA\u0026gt;false\u0026lt;/EnableLUA\u0026gt; \u0026lt;/component\u0026gt; \u0026lt;/settings\u0026gt; \u0026lt;settings pass=\u0026#34;generalize\u0026#34;\u0026gt; \u0026lt;component name=\u0026#34;Microsoft-Windows-Security-SPP\u0026#34; processorArchitecture=\u0026#34;amd64\u0026#34; publicKeyToken=\u0026#34;31bf3856ad364e35\u0026#34; language=\u0026#34;neutral\u0026#34; versionScope=\u0026#34;nonSxS\u0026#34;\u0026gt; \u0026lt;SkipRearm\u0026gt;1\u0026lt;/SkipRearm\u0026gt; \u0026lt;/component\u0026gt; \u0026lt;/settings\u0026gt; \u0026lt;settings pass=\u0026#34;specialize\u0026#34;\u0026gt; \u0026lt;component name=\u0026#34;Microsoft-Windows-Deployment\u0026#34; processorArchitecture=\u0026#34;amd64\u0026#34; publicKeyToken=\u0026#34;31bf3856ad364e35\u0026#34; language=\u0026#34;neutral\u0026#34; versionScope=\u0026#34;nonSxS\u0026#34;\u0026gt; \u0026lt;RunSynchronous\u0026gt; \u0026lt;RunSynchronousCommand wcm:action=\u0026#34;add\u0026#34;\u0026gt; \u0026lt;WillReboot\u0026gt;Always\u0026lt;/WillReboot\u0026gt; \u0026lt;Path\u0026gt;%SystemRoot%\\System32\\reg.exe ADD \u0026#34;HKLM\\System\\CurrentControlSet\\Control\\TimeZoneInformation\u0026#34; /v RealTimeIsUniversal /d 1 /t REG_DWORD /f\u0026lt;/Path\u0026gt; \u0026lt;Order\u0026gt;1\u0026lt;/Order\u0026gt; \u0026lt;/RunSynchronousCommand\u0026gt; \u0026lt;RunSynchronousCommand wcm:action=\u0026#34;add\u0026#34;\u0026gt; \u0026lt;WillReboot\u0026gt;Always\u0026lt;/WillReboot\u0026gt; \u0026lt;Path\u0026gt;e:\\setup.exe /s /v \u0026#34;/qb REBOOT=R ADDLOCAL=ALL\u0026#34;\u0026lt;/Path\u0026gt; \u0026lt;Order\u0026gt;2\u0026lt;/Order\u0026gt; \u0026lt;/RunSynchronousCommand\u0026gt; \u0026lt;/RunSynchronous\u0026gt; \u0026lt;/component\u0026gt; \u0026lt;component name=\u0026#34;Microsoft-Windows-International-Core\u0026#34; processorArchitecture=\u0026#34;amd64\u0026#34; publicKeyToken=\u0026#34;31bf3856ad364e35\u0026#34; language=\u0026#34;neutral\u0026#34; versionScope=\u0026#34;nonSxS\u0026#34;\u0026gt; \u0026lt;InputLocale\u0026gt;0409:00000409\u0026lt;/InputLocale\u0026gt; \u0026lt;SystemLocale\u0026gt;en-US\u0026lt;/SystemLocale\u0026gt; \u0026lt;UILanguage\u0026gt;en-US\u0026lt;/UILanguage\u0026gt; \u0026lt;UILanguageFallback\u0026gt;en-US\u0026lt;/UILanguageFallback\u0026gt; \u0026lt;UserLocale\u0026gt;en-US\u0026lt;/UserLocale\u0026gt; \u0026lt;/component\u0026gt; \u0026lt;component name=\u0026#34;Microsoft-Windows-Security-SPP-UX\u0026#34; processorArchitecture=\u0026#34;amd64\u0026#34; publicKeyToken=\u0026#34;31bf3856ad364e35\u0026#34; language=\u0026#34;neutral\u0026#34; versionScope=\u0026#34;nonSxS\u0026#34;\u0026gt; \u0026lt;SkipAutoActivation\u0026gt;true\u0026lt;/SkipAutoActivation\u0026gt; \u0026lt;/component\u0026gt; \u0026lt;component name=\u0026#34;Microsoft-Windows-SQMApi\u0026#34; processorArchitecture=\u0026#34;amd64\u0026#34; publicKeyToken=\u0026#34;31bf3856ad364e35\u0026#34; language=\u0026#34;neutral\u0026#34; versionScope=\u0026#34;nonSxS\u0026#34;\u0026gt; \u0026lt;CEIPEnabled\u0026gt;0\u0026lt;/CEIPEnabled\u0026gt; \u0026lt;/component\u0026gt; \u0026lt;component name=\u0026#34;Microsoft-Windows-Shell-Setup\u0026#34; processorArchitecture=\u0026#34;amd64\u0026#34; publicKeyToken=\u0026#34;31bf3856ad364e35\u0026#34; language=\u0026#34;neutral\u0026#34; versionScope=\u0026#34;nonSxS\u0026#34;\u0026gt; \u0026lt;ComputerName /\u0026gt; \u0026lt;ProductKey\u0026gt;VDYBN-27WPP-V4HQT-9VMD4-VMK7H\u0026lt;/ProductKey\u0026gt; \u0026lt;/component\u0026gt; \u0026lt;/settings\u0026gt; \u0026lt;settings pass=\u0026#34;oobeSystem\u0026#34;\u0026gt; \u0026lt;component name=\u0026#34;Microsoft-Windows-Shell-Setup\u0026#34; processorArchitecture=\u0026#34;amd64\u0026#34; publicKeyToken=\u0026#34;31bf3856ad364e35\u0026#34; language=\u0026#34;neutral\u0026#34; versionScope=\u0026#34;nonSxS\u0026#34;\u0026gt; \u0026lt;AutoLogon\u0026gt; \u0026lt;Password\u0026gt; \u0026lt;Value\u0026gt;VMware123!\u0026lt;/Value\u0026gt; \u0026lt;!-- Update this --\u0026gt; \u0026lt;PlainText\u0026gt;true\u0026lt;/PlainText\u0026gt; \u0026lt;/Password\u0026gt; \u0026lt;Enabled\u0026gt;true\u0026lt;/Enabled\u0026gt; \u0026lt;Username\u0026gt;Administrator\u0026lt;/Username\u0026gt; \u0026lt;/AutoLogon\u0026gt; \u0026lt;FirstLogonCommands\u0026gt; \u0026lt;SynchronousCommand wcm:action=\u0026#34;add\u0026#34;\u0026gt; \u0026lt;Order\u0026gt;1\u0026lt;/Order\u0026gt; \u0026lt;Description\u0026gt;Set Execution Policy 64 Bit\u0026lt;/Description\u0026gt; \u0026lt;CommandLine\u0026gt;cmd.exe /c powershell -Command \u0026#34;Set-ExecutionPolicy -ExecutionPolicy RemoteSigned -Force\u0026#34;\u0026lt;/CommandLine\u0026gt; \u0026lt;RequiresUserInput\u0026gt;true\u0026lt;/RequiresUserInput\u0026gt; \u0026lt;/SynchronousCommand\u0026gt; \u0026lt;SynchronousCommand wcm:action=\u0026#34;add\u0026#34;\u0026gt; \u0026lt;Order\u0026gt;2\u0026lt;/Order\u0026gt; \u0026lt;Description\u0026gt;Set Execution Policy 32 Bit\u0026lt;/Description\u0026gt; \u0026lt;CommandLine\u0026gt;%SystemDrive%\\Windows\\SysWOW64\\cmd.exe /c powershell -Command \u0026#34;Set-ExecutionPolicy -ExecutionPolicy RemoteSigned -Force\u0026#34;\u0026lt;/CommandLine\u0026gt; \u0026lt;RequiresUserInput\u0026gt;true\u0026lt;/RequiresUserInput\u0026gt; \u0026lt;/SynchronousCommand\u0026gt; \u0026lt;SynchronousCommand wcm:action=\u0026#34;add\u0026#34;\u0026gt; \u0026lt;CommandLine\u0026gt;%SystemRoot%\\System32\\reg.exe ADD HKCU\\SOFTWARE\\Microsoft\\Windows\\CurrentVersion\\Explorer\\Advanced\\ /v HideFileExt /t REG_DWORD /d 0 /f\u0026lt;/CommandLine\u0026gt; \u0026lt;Order\u0026gt;3\u0026lt;/Order\u0026gt; \u0026lt;Description\u0026gt;Show file extensions in Explorer\u0026lt;/Description\u0026gt; \u0026lt;/SynchronousCommand\u0026gt; \u0026lt;SynchronousCommand wcm:action=\u0026#34;add\u0026#34;\u0026gt; \u0026lt;CommandLine\u0026gt;%SystemRoot%\\System32\\reg.exe ADD HKCU\\Console /v QuickEdit /t REG_DWORD /d 1 /f\u0026lt;/CommandLine\u0026gt; \u0026lt;Order\u0026gt;4\u0026lt;/Order\u0026gt; \u0026lt;Description\u0026gt;Enable QuickEdit mode\u0026lt;/Description\u0026gt; \u0026lt;/SynchronousCommand\u0026gt; \u0026lt;SynchronousCommand wcm:action=\u0026#34;add\u0026#34;\u0026gt; \u0026lt;CommandLine\u0026gt;%SystemRoot%\\System32\\reg.exe ADD HKCU\\SOFTWARE\\Microsoft\\Windows\\CurrentVersion\\Explorer\\Advanced\\ /v Start_ShowRun /t REG_DWORD /d 1 /f\u0026lt;/CommandLine\u0026gt; \u0026lt;Order\u0026gt;5\u0026lt;/Order\u0026gt; \u0026lt;Description\u0026gt;Show Run command in Start Menu\u0026lt;/Description\u0026gt; \u0026lt;/SynchronousCommand\u0026gt; \u0026lt;SynchronousCommand wcm:action=\u0026#34;add\u0026#34;\u0026gt; \u0026lt;CommandLine\u0026gt;%SystemRoot%\\System32\\reg.exe ADD HKCU\\SOFTWARE\\Microsoft\\Windows\\CurrentVersion\\Explorer\\Advanced\\ /v StartMenuAdminTools /t REG_DWORD /d 1 /f\u0026lt;/CommandLine\u0026gt; \u0026lt;Order\u0026gt;6\u0026lt;/Order\u0026gt; \u0026lt;Description\u0026gt;Show Administrative Tools in Start Menu\u0026lt;/Description\u0026gt; \u0026lt;/SynchronousCommand\u0026gt; \u0026lt;SynchronousCommand wcm:action=\u0026#34;add\u0026#34;\u0026gt; \u0026lt;CommandLine\u0026gt;%SystemRoot%\\System32\\reg.exe ADD HKLM\\SYSTEM\\CurrentControlSet\\Control\\Power\\ /v HibernateFileSizePercent /t REG_DWORD /d 0 /f\u0026lt;/CommandLine\u0026gt; \u0026lt;Order\u0026gt;7\u0026lt;/Order\u0026gt; \u0026lt;Description\u0026gt;Zero Hibernation File\u0026lt;/Description\u0026gt; \u0026lt;/SynchronousCommand\u0026gt; \u0026lt;SynchronousCommand wcm:action=\u0026#34;add\u0026#34;\u0026gt; \u0026lt;CommandLine\u0026gt;%SystemRoot%\\System32\\reg.exe ADD HKLM\\SYSTEM\\CurrentControlSet\\Control\\Power\\ /v HibernateEnabled /t REG_DWORD /d 0 /f\u0026lt;/CommandLine\u0026gt; \u0026lt;Order\u0026gt;8\u0026lt;/Order\u0026gt; \u0026lt;Description\u0026gt;Disable Hibernation Mode\u0026lt;/Description\u0026gt; \u0026lt;/SynchronousCommand\u0026gt; \u0026lt;SynchronousCommand wcm:action=\u0026#34;add\u0026#34;\u0026gt; \u0026lt;CommandLine\u0026gt;cmd.exe /c wmic useraccount where \u0026#34;name=\u0026#39;Administrator\u0026#39;\u0026#34; set PasswordExpires=FALSE\u0026lt;/CommandLine\u0026gt; \u0026lt;Order\u0026gt;9\u0026lt;/Order\u0026gt; \u0026lt;Description\u0026gt;Disable password expiration for Administrator user\u0026lt;/Description\u0026gt; \u0026lt;/SynchronousCommand\u0026gt; \u0026lt;SynchronousCommand wcm:action=\u0026#34;add\u0026#34;\u0026gt; \u0026lt;CommandLine\u0026gt;cmd.exe /c %SystemDrive%\\Windows\\System32\\WindowsPowerShell\\v1.0\\powershell.exe -File a:\\enable-winrm.ps1\u0026lt;/CommandLine\u0026gt; \u0026lt;Description\u0026gt;Enable WinRM\u0026lt;/Description\u0026gt; \u0026lt;Order\u0026gt;10\u0026lt;/Order\u0026gt; \u0026lt;/SynchronousCommand\u0026gt; \u0026lt;SynchronousCommand wcm:action=\u0026#34;add\u0026#34;\u0026gt; \u0026lt;CommandLine\u0026gt;cmd.exe /c a:\\disable-network-discovery.cmd\u0026lt;/CommandLine\u0026gt; \u0026lt;Description\u0026gt;Disable Network Discovery\u0026lt;/Description\u0026gt; \u0026lt;Order\u0026gt;11\u0026lt;/Order\u0026gt; \u0026lt;/SynchronousCommand\u0026gt; \u0026lt;SynchronousCommand wcm:action=\u0026#34;add\u0026#34;\u0026gt; \u0026lt;!-- Customization for public network issue --\u0026gt; \u0026lt;Order\u0026gt;12\u0026lt;/Order\u0026gt; \u0026lt;Description\u0026gt;Configure additional policy for public network\u0026lt;/Description\u0026gt; \u0026lt;CommandLine\u0026gt;cmd.exe /c powershell -NoProfile -ExecutionPolicy Bypass -Command \u0026#34;Set-NetFirewallRule -Name \u0026#39;WINRM-HTTP-In-TCP-PUBLIC\u0026#39; -RemoteAddress Any; Set-Item -Path \u0026#39;WSMan:\\localhost\\Service\\Auth\\Basic\u0026#39; -Value $true; Set-Item -Path \u0026#39;WSMan:\\localhost\\Service\\AllowUnencrypted\u0026#39; -Value $true\u0026#34;\u0026lt;/CommandLine\u0026gt; \u0026lt;/SynchronousCommand\u0026gt; \u0026lt;/FirstLogonCommands\u0026gt; \u0026lt;OOBE\u0026gt; \u0026lt;HideEULAPage\u0026gt;true\u0026lt;/HideEULAPage\u0026gt; \u0026lt;HideLocalAccountScreen\u0026gt;true\u0026lt;/HideLocalAccountScreen\u0026gt; \u0026lt;HideOEMRegistrationScreen\u0026gt;true\u0026lt;/HideOEMRegistrationScreen\u0026gt; \u0026lt;HideOnlineAccountScreens\u0026gt;true\u0026lt;/HideOnlineAccountScreens\u0026gt; \u0026lt;HideWirelessSetupInOOBE\u0026gt;true\u0026lt;/HideWirelessSetupInOOBE\u0026gt; \u0026lt;NetworkLocation\u0026gt;Work\u0026lt;/NetworkLocation\u0026gt; \u0026lt;ProtectYourPC\u0026gt;1\u0026lt;/ProtectYourPC\u0026gt; \u0026lt;SkipMachineOOBE\u0026gt;true\u0026lt;/SkipMachineOOBE\u0026gt; \u0026lt;SkipUserOOBE\u0026gt;true\u0026lt;/SkipUserOOBE\u0026gt; \u0026lt;/OOBE\u0026gt; \u0026lt;RegisteredOrganization\u0026gt;Organization\u0026lt;/RegisteredOrganization\u0026gt; \u0026lt;RegisteredOwner\u0026gt;Owner\u0026lt;/RegisteredOwner\u0026gt; \u0026lt;DisableAutoDaylightTimeSet\u0026gt;false\u0026lt;/DisableAutoDaylightTimeSet\u0026gt; \u0026lt;TimeZone\u0026gt;Pacific Standard Time\u0026lt;/TimeZone\u0026gt; \u0026lt;UserAccounts\u0026gt; \u0026lt;AdministratorPassword\u0026gt; \u0026lt;Value\u0026gt;VMware123!\u0026lt;/Value\u0026gt; \u0026lt;!-- Update this --\u0026gt; \u0026lt;PlainText\u0026gt;true\u0026lt;/PlainText\u0026gt; \u0026lt;/AdministratorPassword\u0026gt; \u0026lt;LocalAccounts\u0026gt; \u0026lt;LocalAccount wcm:action=\u0026#34;add\u0026#34;\u0026gt; \u0026lt;Description\u0026gt;Administrator\u0026lt;/Description\u0026gt; \u0026lt;DisplayName\u0026gt;Administrator\u0026lt;/DisplayName\u0026gt; \u0026lt;Group\u0026gt;Administrators\u0026lt;/Group\u0026gt; \u0026lt;Name\u0026gt;Administrator\u0026lt;/Name\u0026gt; \u0026lt;/LocalAccount\u0026gt; \u0026lt;LocalAccount wcm:action=\u0026#34;add\u0026#34;\u0026gt; \u0026lt;Password\u0026gt; \u0026lt;Value\u0026gt;VMware123!\u0026lt;/Value\u0026gt; \u0026lt;!-- Update this --\u0026gt; \u0026lt;PlainText\u0026gt;true\u0026lt;/PlainText\u0026gt; \u0026lt;/Password\u0026gt; \u0026lt;Description\u0026gt;For log collection\u0026lt;/Description\u0026gt; \u0026lt;DisplayName\u0026gt;Admin Account\u0026lt;/DisplayName\u0026gt; \u0026lt;Name\u0026gt;WindowsAdmin\u0026lt;/Name\u0026gt; \u0026lt;Group\u0026gt;Administrators\u0026lt;/Group\u0026gt; \u0026lt;/LocalAccount\u0026gt; \u0026lt;/LocalAccounts\u0026gt; \u0026lt;/UserAccounts\u0026gt; \u0026lt;/component\u0026gt; \u0026lt;/settings\u0026gt; \u0026lt;/unattend\u0026gt; Update the vsphere packer variables in packer-variables/vsphere.j2 here is an example file, the fields that must be updates are marked with a comment {# Update this #}. { {# vCenter server IP or FQDN #} \u0026#34;vcenter_server\u0026#34;:\u0026#34;vcsa9-wld.vcf.lab\u0026#34;, {# Update this #} {# vCenter username #} \u0026#34;username\u0026#34;:\u0026#34;administrator@vcf9-wld.local\u0026#34;, {# Update this #} {# vCenter user password #} \u0026#34;password\u0026#34;:\u0026#34;VMware123!VMware123!\u0026#34;, {# Update this #} {# Datacenter name where packer creates the VM for customization #} \u0026#34;datacenter\u0026#34;:\u0026#34;wld9-DC\u0026#34;, {# Update this #} {# Datastore name for the VM #} \u0026#34;datastore\u0026#34;:\u0026#34;cls-wld9-vsan01\u0026#34;, {# Update this #} {# [Optional] Folder name #} \u0026#34;folder\u0026#34;:\u0026#34;\u0026#34;, {# Cluster name where packer creates the VM for customization #} \u0026#34;cluster\u0026#34;: \u0026#34;wls-wld9\u0026#34;, {# Update this #} {# Packer VM network #} \u0026#34;network\u0026#34;: \u0026#34;/wld9-DC/network/Virtual Private Clouds/image-builder-3/vm-public/vm-public\u0026#34;, {# Update this #} {# To use insecure connection with vCenter #} \u0026#34;insecure_connection\u0026#34;: \u0026#34;true\u0026#34;, {# TO create a clone of the Packer VM after customization#} \u0026#34;linked_clone\u0026#34;: \u0026#34;true\u0026#34;, {# To create a snapshot of the Packer VM after customization #} \u0026#34;create_snapshot\u0026#34;: \u0026#34;true\u0026#34;, {# To destroy Packer VM after Image Build is completed #} \u0026#34;destroy\u0026#34;: \u0026#34;true\u0026#34; } Update the windows specific packer vars. There are two files packer-variables/windows/default-args-windows.j2 and packer-variables/windows/vsphere-windows.j2 . Below are sample files with comments on where to update. vsphere-windows.j2 - update the paths with the output from the iso upload script\n{ {# [Optional] Windows only: Windows OS Image #} \u0026#34;os_iso_path\u0026#34;: \u0026#34;[cls-wld9-vsan01] isos/en-us_windows_server_2022_x64_dvd_620d7eac.iso\u0026#34;, {# Update this #} {# [Optional] Windows only: VMware Tools Image #} \u0026#34;vmtools_iso_path\u0026#34;: \u0026#34;[cls-wld9-vsan01] isos/vmtools-windows.iso\u0026#34; {# Update this #} } default-args-windows.j2 - all that needs to be added here is the windows_admin_password\n{ \u0026#34;additional_executables_destination_path\u0026#34;: \u0026#34;C:\\\\ProgramData\\\\Temp\u0026#34;, \u0026#34;additional_executables_list\u0026#34;: \u0026#34;http://{{ host_ip }}:{{ artifacts_container_port }}/artifacts/{{ kubernetes_version }}/bin/windows/amd64/registry.exe,http://{{ host_ip }}:{{ artifacts_container_port }}/artifacts/{{ kubernetes_version }}/bin/windows/amd64/goss.exe\u0026#34;, \u0026#34;additional_executables\u0026#34;: \u0026#34;true\u0026#34;, \u0026#34;additional_url_images\u0026#34;: \u0026#34;false\u0026#34;, \u0026#34;additional_url_images_list\u0026#34;: \u0026#34;\u0026#34;, \u0026#34;additional_prepull_images\u0026#34;: \u0026#34;\u0026#34;, \u0026#34;build_version\u0026#34;: \u0026#34;{{ os_type }}-kube-{{ kubernetes_series }}-{{ ova_ts_suffix }}\u0026#34;, \u0026#34;cloudbase_init_url\u0026#34;: \u0026#34;http://{{ host_ip }}:{{ artifacts_container_port }}/artifacts/{{ kubernetes_version }}/bin/windows/amd64/CloudbaseInitSetup_x64.msi\u0026#34;, \u0026#34;cloudbase_real_time_clock_utc\u0026#34;: \u0026#34;true\u0026#34;, \u0026#34;containerd_url\u0026#34;: \u0026#34;http://{{ host_ip }}:{{ artifacts_container_port }}/artifacts/{{ kubernetes_version }}/bin/windows/amd64/cri-containerd.tar\u0026#34;, \u0026#34;containerd_sha256_windows\u0026#34;: \u0026#34;{{ containerd_sha256_windows_amd64 }}\u0026#34;, \u0026#34;containerd_version\u0026#34;: \u0026#34;{{ containerd }}\u0026#34;, \u0026#34;convert_to_template\u0026#34;: \u0026#34;true\u0026#34;, \u0026#34;create_snapshot\u0026#34;: \u0026#34;false\u0026#34;, \u0026#34;disable_hypervisor\u0026#34;: \u0026#34;false\u0026#34;, \u0026#34;disk_size\u0026#34;: \u0026#34;40960\u0026#34;, \u0026#34;kubernetes_base_url\u0026#34;: \u0026#34;http://{{ host_ip }}:{{ artifacts_container_port }}/artifacts/{{ kubernetes_version }}/bin/windows/amd64\u0026#34;, \u0026#34;kubernetes_series\u0026#34;: \u0026#34;{{ kubernetes_series }}\u0026#34;, \u0026#34;kubernetes_semver\u0026#34;: \u0026#34;{{ kubernetes_version }}\u0026#34;, \u0026#34;kubernetes_typed_version\u0026#34;: \u0026#34;{{ image_version }}\u0026#34;, \u0026#34;load_additional_components\u0026#34;: \u0026#34;true\u0026#34;, \u0026#34;netbios_host_name_compatibility\u0026#34;: \u0026#34;false\u0026#34;, \u0026#34;nssm_url\u0026#34;: \u0026#34;http://{{ host_ip }}:{{ artifacts_container_port }}/artifacts/{{ kubernetes_version }}/bin/windows/amd64/nssm.exe\u0026#34;, \u0026#34;prepull\u0026#34;: \u0026#34;false\u0026#34;, \u0026#34;pause_image\u0026#34;: \u0026#34;localhost:5000/vmware.io/pause:{{ pause }}\u0026#34;, \u0026#34;runtime\u0026#34;: \u0026#34;containerd\u0026#34;, \u0026#34;template\u0026#34;: \u0026#34;\u0026#34;, \u0026#34;unattend_timezone\u0026#34;: \u0026#34;Pacific Standard Time\u0026#34;, \u0026#34;windows_updates_categories\u0026#34;: \u0026#34;\u0026#34;, \u0026#34;windows_updates_kbs\u0026#34;: \u0026#34;\u0026#34;, \u0026#34;wins_url\u0026#34;: \u0026#34;\u0026#34;, \u0026#34;custom_role\u0026#34;: \u0026#34;true\u0026#34;, \u0026#34;custom_role_names\u0026#34;: \u0026#34;/image-builder/images/capi/image/ansible-windows\u0026#34;, \u0026#34;ansible_user_vars\u0026#34;: \u0026#34;ansible_winrm_read_timeout_sec=600 ansible_winrm_operation_timeout_sec=590 artifacts_container_url=http://{{ host_ip }}:{{ artifacts_container_port }} imageVersion={{ image_version|replace(\u0026#39;-\u0026#39;, \u0026#39;.\u0026#39;) }} registry_store_archive_url=http://{{ host_ip }}:{{ artifacts_container_port }}/artifacts/{{ kubernetes_version }}/registries/{{ registry_store_path }}\u0026#34;, \u0026#34;vmx_version\u0026#34;: \u0026#34;21\u0026#34;, \u0026#34;debug_tools\u0026#34;: \u0026#34;false\u0026#34;, \u0026#34;enable_auto_kubelet_service_restart\u0026#34;: \u0026#34;false\u0026#34;, \u0026#34;windows_admin_password\u0026#34;: \u0026#34;VMware123!\u0026#34; {# Update this #} } Running the Build # Now that all of the specific settings are in place we can run the build. This will output an OVA that can then be uploaded to a content lib and used in a VKS cluster.\nStart the artifacts container and run the build. The TKR_SUFFIX should match the suffix of your current linux VKR versions, this used when resolving the correct ova from the content lib. The HOST_IP should be your workstation IP that this command is running from. The IMAGE_ARTIFACTS_PATH is where you want the OVA to be created. The AUTO_UNATTEND_ANSWER_FILE_PATH is the path to your answers file. ##start the artifacts container make run-artifacts-container ARTIFACTS_CONTAINER_PORT=8081 ##run this from the vks-image-builder directory make build-node-image OS_TARGET=windows-2022-efi TKR_SUFFIX=vkr.4 HOST_IP=10.0.0.180 IMAGE_ARTIFACTS_PATH=/home/will/windows-build-2/image ARTIFACTS_CONTAINER_PORT=8081 PACKER_HTTP_PORT=8082 AUTO_UNATTEND_ANSWER_FILE_PATH=/home/will/windows-build-2/vks-image-builder/windows_autounattend.xml Create a content lib and upload the OVA. ## update the datastore with your datastore as well as the path to the ova govc library.create -ds \u0026#34;cls-wld9-vsan01\u0026#34; \u0026#34;windows-vkrs\u0026#34; govc library.import windows-vkrs ./image/ovas/windows-2022-amd64-v1.34.1---vmware.1-vkr.4.ova Setting up the content library # Before deploying a cluster, we need to associate our new content library with the supervisor. We can do this by going to the supervisor settings and under general adding another content library.\n💡 Note: when you do this there is a warning about multiple content libraries and needing to disambiguate between them. depending on how you build clusters you may need to add the correct annoations to specify the content library. This doc has more details. Verify that the image is showing up in the supervisor. Run this from the supervisor cluster context.\nk get osimages -A | grep windows vmi-4aacfc6e370e28a1e v1.34.1+vmware.1 windows 2022 amd64 cvmi 13s k get cclitem -A | grep windows clitem-4aacfc6e370e28a1e windows-2022-amd64-v1.34.1---vmware.1-vkr.4 cl-c80585866b93825a5 OVF True true 20615149892 true 100s Take note of the content library ID for future use in cluster builds.\nDeploy a cluster # Now that we have verified the image is available we can deploy a cluster. There are a number of ways to do this but for the purpose of this post I am just going to share the yaml for the cluster and you can deploy it how you would like. The main thing to note below is the Windows node poo along with the annotation that tells it which content library to use.\napiVersion: cluster.x-k8s.io/v1beta1 kind: Cluster metadata: name: windows namespace: dev-ts6hw spec: clusterNetwork: pods: cidrBlocks: - 192.168.156.0/20 services: cidrBlocks: - 10.96.0.0/12 serviceDomain: cluster.local topology: class: builtin-generic-v3.5.0 classNamespace: vmware-system-vks-public version: v1.34.1---vmware.1-vkr.4 variables: - name: kubernetes value: certificateRotation: enabled: true renewalDaysBeforeExpiry: 90 - name: vmClass value: best-effort-small - name: storageClass value: vsan-default-storage-policy controlPlane: replicas: 1 metadata: annotations: run.tanzu.vmware.com/resolve-os-image: os-name=photon, content-library=cl-a3b5e06b12f2737ca workers: machineDeployments: - class: node-pool name: windows-np-j72b replicas: 1 variables: overrides: - name: vmClass value: best-effort-xsmall - class: node-pool name: windows-nodepool-l7rs replicas: 1 metadata: annotations: run.tanzu.vmware.com/resolve-os-image: os-type=windows, content-library=cl-c80585866b93825a5 variables: overrides: - name: vmClass value: best-effort-large Customizing images with Ansible # Sometimes it may be necessary to modify the image with custom scripts or binaries. This should be used with caution and also should only be used for things that cannot be done through a native K8s operator during runtime. Also this should not be used for anything that would require embedding passwords into the OVA. With those caveats out of the way let’s add some Ansible to modify the image build.\nIn this example we will add a simple Ansible task that runs a powershell script. This script does not perform any meaningful actions in this example but provides the details on how to set this up so that you can use a similar process for anything you need to install.\nAdd a new file to the vks-image-builder/ansible-windows/tasks folder. This ansible-windows folder is the Ansible role that executes by default during the build process. touch vks-image-builder/ansible-windows/tasks/exec-pwsh.yml Add the contents of the Ansible task to the new file . - name: Execute custom BYOI script ansible.builtin.script: scripts/helloworld.ps1 Add the powershell script to the files in the Ansible role. touch vks-image-builder/ansible-windows/files/scripts/helloworld.ps1 Add the contents of the script to the file. Write-Output \u0026#34;Hello, World!\u0026#34; Update the main.yml to include your new tasks - import_tasks: exec-pwsh.yml Updating cluster nodes for patching # Since these are images that you maintain, you may need to update an image with a one off patch etc. This may also be the case for images that don’t have a new k8s version to go along with it. In the case of patching a with new K8s version it’s pretty simple, you build a new image with the new K8s version and then update the K8s version on your cluster definition. But what about the case where there are no changes upstream from Broadcom, that’s what we will cover here. In this case we want to patch or change something on the image and want to roll that out to our clusters without changing anything else.\nTo do this, we need to understand.a bit about how an OVA in a content library is resolved by VKS.\nThe VKR version selected in the cluster definition. This looks something like v1.34.1---vmware.1-vkr.4, we can see that this is version 1.34.1 of K8s, also that it’s a patch release of vmware.1-vkr.4 this suffix is used during the image resolution process.\nThe OS , this is set on the node pools usually Photon, Ubuntu, or Windows.\nThe OS version, for Ubuntu we might see 22.04, Windows we would see 2022\nThe content library setting\nWhen these are combined VKS looks at the available OS images that match the right VKR version and OS settings and then pulls the right ova from the content library to create the node.\nNow the challenge we run into is that if we just update our patch suffix to something like vkr.5 and then when we build a cluster the Windows nodes would be resolved but if our Linux nodes don’t also have a vkr.5 patch release then it won’t find the right images. We also need to make sure our content library item is following the standard naming convention so that it is picked up bu VKS, the name geenrated by the image builder creates this in the proper format(windows-2022-amd64-v1.34.1---vmware.1-vkr.4). Since there is already an OVA in the content library named that we can’t upload the new one unless we want to overwrite it. Overwriting the image is a perfectly accetable way to roll out a new patch however I would prefer to have some more control over the process. So we need another way to manage this. This is where the content library setting comes in, we can create a new content library for our patch and upload the new OVA then selectively roll out the new node image.\nCreate a new content library for the patch and upload the new image. govc library.create -ds \u0026#34;cls-wld9-vsan01\u0026#34; \u0026#34;windows-patch-12626\u0026#34; govc library.import windows-patch-12626 ./image/ovas/windows-2022-amd64-v1.34.1---vmware.1-vkr.4.ova Update supervisor config to add the new content library Update your cluster yaml to use the new content library ID. k get cclitem -A | grep -i windows run.tanzu.vmware.com/resolve-os-image: os-type=windows, content-library=cl-1c5b6ba4a0e41aa16 ","date":"27 January 2026","externalUrl":null,"permalink":"/posts/windows-vks/","section":"Posts","summary":"There have been numerous how-to guides over the past few years on building and deploying Windows clusters on the different Kubernetes distributions supported by VMware. This post aims to solidify the understanding of the current most up to date process of doing this on vSphere Kubernetes Service which is the definitive Kubernetes service for VMware. At the time of writing the latest vSphere Kubernetes Release(VKR) is v1.34.1---vmware.1-vkr.4 so that is what this will be based on.\n","title":"Deploying Windows Clusters on vSphere Kubernetes Service with VKS Image Builder","type":"posts"},{"content":"","date":"27 January 2026","externalUrl":null,"permalink":"/tags/vks/","section":"Tags","summary":"","title":"Vks","type":"tags"},{"content":"","date":"27 January 2026","externalUrl":null,"permalink":"/tags/vsphere/","section":"Tags","summary":"","title":"Vsphere","type":"tags"},{"content":"","date":"27 January 2026","externalUrl":null,"permalink":"/tags/windows/","section":"Tags","summary":"","title":"Windows","type":"tags"},{"content":"When deploying VKS clusters, VMs, or Pods that use extra volumes in VCF with the Supervisor, the volumes are managed through PVCs and in turn PVs. These PVs create CNS(Cloud Native Storage) volumes which integrate vSphere and Kubernetes to enable the creation and management of container volumes in a vSphere environment. This is an extremely common thing to do when deploying a VKS cluster for example , you usually add additional volumes to the nodes for additional storage of images( var/lib/containerd). This additional volume is managed through a PVC and ultimately a PV.\nOne challenge with this from an administrator point of view is getting insight into all of the volumes and what they are being used for and where. vCenter does have a native UI to explore these and filter them and govc can also be used to query them with the CLI, however many times I want to see these volumes from a namespace and PVC based view. It can also be challenging to associate the volumes with the PVC when looking through the raw details from the govc output. That’s why I created this simple python script to help with getting the details for volumes based on a Supervisor namespace.\nThe script used both kubectl output and govc output and correlates the PVCs to the underlying CNS volumes, it then returns relevant details about the volumes as well as sorts them between node volumes and volumes that are being used in the clusters for workload PVCs.\nExample Usage # let’s say I have a supervisor namespace called gitops-932tv and in that namespace I have a VKS cluster deployed. Also in that cluster I have an app deployed that uses a number of persistent disks. As an administrator I may need to determine where these volumes are located on my datastores and also get some details like there volume IDs etc. in case I need to relocate them. To get this info I can simple run the following.\nSetup the kube context and govc details\n# sets the kube context vcf context use \u0026lt;supervisor-conext\u0026gt; export GOVC_INSECURE=true export GOVC_PASSWORD=\u0026#39;password\u0026#39; export GOVC_USERNAME=administrator@vsphere.local export GOVC_URL=https://vcsa.myorg.com clone the repo and run the script\ngit clone https://github.com/warroyo/cns-tooling cd cns-tooling/pvc-audit python3 vks_disk_audit.py gitops-932tv With those two steps the output will look similar to this. You will notice that the we get two separate sections, one for node volumes and one for in cluster volumes. The node volumes section shows which node the volumes are attached to as well as details about the size, cluster, datastore etc. For the in cluster volumes, these are typically PVCs used by workloads in the cluster, these show similar details but also include the referred entity metadata about the PVC name in the cluster and which pod is using it.\n--- Starting Audit for Namespace: gitops-932tv --- Mapping VSphereMachines to Clusters... Mapped 2 nodes across clusters. Querying all PVCs in namespace... Found 7 PVC(s). Resolving PV handles... Querying vSphere CNS for 7 volumes... ======================================================================================= NODE VOLUMES (Attached) ======================================================================================= PVC Name Node Cluster Volume Name Volume ID Datastore Capacity Referred Entity ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ tfd-1-8q26s-ql69w-vol-b9xl tfd-1-8q26s-ql69w tfd-1 pvc-9e7d6921-d573-4203-ad8a-9e420f490446 560fd41e-f243-4c96-997e-8bf7b7996e95 vsan:8740804e6737443c-b388c53757aaaf93/ 20.00 GB - tfd-1-tfd-1-jrxdt-pmhmn-cv4hj-vol-b9xl tfd-1-tfd-1-jrxdt-pmhmn-cv4hj tfd-1 pvc-2417b407-2bed-4cdb-88fb-7b1ebf6653ad ecfffa9d-483c-4fe1-a391-e4fe1986e52d vsan:8740804e6737443c-b388c53757aaaf93/ 20.00 GB - ======================================================================================= IN-CLUSTER PVCs ======================================================================================= PVC Name Cluster Volume Name Volume ID Datastore Capacity Referred Entity ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ f478b761-3832-4c56-8cc7-10b44e5a968b-1fb60ee1-66e8-4ce6-97a6-c7cb12070bdf tfd-1 pvc-6b57785d-828b-4f1d-ab32-7c617e60a908 2b2deeab-53cf-4551-bba4-e787dc1567d8 vsan:8740804e6737443c-b388c53757aaaf93/ 1.00 GB PVC:music-store/order-pvc, Pod:order-service-69987cbbfd-mlj2c f478b761-3832-4c56-8cc7-10b44e5a968b-232680b1-d8d7-4eaa-9e50-66a12bb0fb0a tfd-1 pvc-788a9fa5-6db8-4151-ad5b-3f32c5dcda16 b2c6ac20-b002-4f0a-948b-4fccef7d8c3a vsan:8740804e6737443c-b388c53757aaaf93/ 1.00 GB PVC:music-store/cart-pvc, Pod:cart-service-7cc8794c86-x2m6v f478b761-3832-4c56-8cc7-10b44e5a968b-3519cba5-07ba-45a5-b329-e647a3500c34 tfd-1 pvc-b7ec835b-ef13-4b56-9fae-4f6b2c6395ce 5fd5d74c-2b77-4ecb-ae95-c4a22ae99111 vsan:8740804e6737443c-b388c53757aaaf93/ 1.00 GB Pod:postgres-bcf8997c4-89pkj, PVC:music-store/postgres-pvc f478b761-3832-4c56-8cc7-10b44e5a968b-7f7b7933-d67a-479c-8197-47c61708f1c6 tfd-1 pvc-92f87c6b-cfdc-4377-8a79-11d6dd6b3b1a 6fe37d12-9c30-4ab5-aaf7-7f9276e3ba49 vsan:8740804e6737443c-b388c53757aaaf93/ 1.00 GB PVC:music-store/music-store-1-pvc f478b761-3832-4c56-8cc7-10b44e5a968b-d1d19249-7cb0-4c7d-914e-ec866e998aaa tfd-1 pvc-06cbb085-57c3-4b27-856a-713b45e0c53b 0f28021c-c85c-4f36-b9de-faa26fedb232 vsan:8740804e6737443c-b388c53757aaaf93/ 1.00 GB PVC:music-store/users-pvc, Pod:users-service-855678b958-9zcdb --- Audit Complete --- ","date":"6 January 2026","externalUrl":null,"permalink":"/posts/auditing-cns-volumes-in-vcf/","section":"Posts","summary":"Learn how to effectively audit CNS volumes in VCF using a Python script to gain insights into volumes from a namespace-based view","title":"Auditing CNS Volumes in VCF","type":"posts"},{"content":"","date":"6 January 2026","externalUrl":null,"permalink":"/tags/cloud-native/","section":"Tags","summary":"","title":"Cloud-Native","type":"tags"},{"content":"","date":"6 January 2026","externalUrl":null,"permalink":"/tags/cns/","section":"Tags","summary":"","title":"Cns","type":"tags"},{"content":"","date":"2 November 2023","externalUrl":null,"permalink":"/tags/tanzu/","section":"Tags","summary":"","title":"Tanzu","type":"tags"},{"content":" Overview # I have had a question come up a few times with customers and coworkers about how to reduce duplication when creating clusters with Tanzu Mission Control(TMC). The question or issue that is usually brought up is that the platform engineering team wants to be able to create clusters quickly and many of the settings between cluster creation are the exact same, thus having a lot of duplication between clusters. When looking at the TMC UI there\u0026rsquo;s not a way to set custom defaults today to be able to remove the need to fill in every field each time you create a cluster. However, using the UI is probably not the approach a platform team wants to take to scale anyway. It\u0026rsquo;s much more efficient to codify the clusters and automate the creation. In this post, we will walk through creating cluster templates and using the Tanzu CLI to create clusters with minimal inputs. We will focus on TKG Clusters mostly, but I will also provide some commands that work with AKS and EKS clusters are well.\nBrief note on the Tanzu CLI # The Tanzu CLI can now be installed through standard package managers or directly via the binary from GitHub, see the install instructions here. The TMC standalone CLI is in the process of being deprecated so you will want to use the new plugins for the Tanzu CLI. They will be installed when you connect the Tanzu CLI to your TMC endpoint.\nTemplating clusters # Getting the base template # The CLI provides a way to get an existing cluster\u0026rsquo;s spec, I always recommend going with this approach over trying to create one from scratch. Before we start templating the cluster, create a cluster through the UI. Then use the below command to pull the cluster\u0026rsquo;s yaml spec back and save it in a file.\n# for TKG tanzu tmc cluster get \u0026lt;cluster-name\u0026gt; -m \u0026lt;mgmt-cluster\u0026gt; -p \u0026lt;provisioner\u0026gt; \u0026gt; template.yml ## for EKS tanzu tmc ekscluster get \u0026lt;cluster-name\u0026gt; -c \u0026lt;credential\u0026gt; -r \u0026lt;region\u0026gt; \u0026gt; template.yml ## for AKS tanzu tmc akscluster get \u0026lt;cluster-name\u0026gt; -c \u0026lt;credential\u0026gt; -r \u0026lt;resource-group\u0026gt; -s \u0026lt;subscription\u0026gt; \u0026gt; template.yml The contents of the above command will look slightly different depending on the k8s provider you are using, but all of them will be a yaml file that describes all of the settings for the cluster.\nRemove any extra fields # Just like when pulling back a resource from a k8s cluster there will be fields that we will want to omit from our template. For example the status section. This will be different between providers, but because we are focused on TKG in this example I have listed the field that I omitted below. Technically many of these fields do not need to be removed since the API will just ignore them, but since we are creating a template and don\u0026rsquo;t want a bunch of extra stuff that might be confusing we will remove anything that is not needed.\nfullname.orgId\nmeta\nlabels\ntmc.cloud.vmware.com/creator annotations\ncreationTime\ngeneration\nresourceVersion\nuid\nupdateTime\nspec.topology.variables\nextensionCert\nuser - if you would like to provide your ssh public key, keep this one.\nclusterEncryptionConfigYaml\nTKR_DATA\nstatus - entire section\nAll of the fields above are generated by TMC when the cluster is created. Your YAML file should now look similar to the one below.\nfullName: managementClusterName: h2o-4-19340 name: tmc-base-template provisionerName: lab meta: labels: example-label: example spec: clusterGroupName: default tmcManaged: true topology: clusterClass: tanzukubernetescluster controlPlane: metadata: annotations: example-cp-annotation: example labels: example-cp-label: example osImage: arch: amd64 name: ubuntu version: \u0026#34;20.04\u0026#34; replicas: 3 network: pods: cidrBlocks: - 172.20.0.0/16 serviceDomain: cluster.local services: cidrBlocks: - 10.96.0.0/16 nodePools: - info: name: md-0 spec: class: node-pool metadata: labels: exmaple-np-label: example osImage: arch: amd64 name: ubuntu version: \u0026#34;20.04\u0026#34; overrides: - name: vmClass value: best-effort-large - name: storageClass value: vc01cl01-t0compute replicas: 2 variables: - name: defaultStorageClass value: vc01cl01-t0compute - name: storageClass value: vc01cl01-t0compute - name: storageClasses value: - vc01cl01-t0compute - name: vmClass value: best-effort-large - name: ntp value: time1.oc.vmware.com version: v1.23.8+vmware.2-tkg.2-zshippable type: kind: TanzuKubernetesCluster package: vmware.tanzu.manage.v1alpha1.managementcluster.provisioner.tanzukubernetescluster version: v1alpha1 Templating with YTT # There are many options for templating files, but since this is a YAML file and we like Carvel YTT that is what is used for this example. I would highly recommend reading up on YTT and testing it out for different use cases, it\u0026rsquo;s a very powerful YAML templating language.\nDetermine the variable fields # first, we need to determine which fields should be variable. This could be any field, but we want to reuse as much as possible as well. These fields are entirely up to you and your needs.\nTemplate the fields with YTT # the YTT docs explain how to use data values in a YAML file. This is what we will be using to template the file. Using the same file from above the below template is what I have come up with.\n#@ load(\u0026#34;@ytt:data\u0026#34;, \u0026#34;data\u0026#34;) fullName: managementClusterName: #@ data.values.mgmt_cluster_name name: #@ data.values.cluster_name provisionerName: #@ data.values.provisioner meta: #@ if/end hasattr( data.values, \u0026#34;cluster_labels\u0026#34;): labels: #@ data.values.cluster_labels spec: clusterGroupName: #@ data.values.cluster_group tmcManaged: true topology: clusterClass: tanzukubernetescluster controlPlane: metadata: #@ if/end hasattr( data.values, \u0026#34;cp_annotations\u0026#34;): annotations: #@ data.values.cp_annotations #@ if/end hasattr( data.values, \u0026#34;cp_labels\u0026#34;): labels: #@ data.values.cluster_labels osImage: arch: amd64 name: ubuntu version: \u0026#34;20.04\u0026#34; replicas: 3 network: pods: cidrBlocks: - 172.20.0.0/16 serviceDomain: cluster.local services: cidrBlocks: - 10.96.0.0/16 nodePools: - info: name: md-0 spec: class: node-pool metadata: #@ if/end hasattr( data.values, \u0026#34;node_labels\u0026#34;): labels: #@ data.values.node_labels osImage: arch: amd64 name: ubuntu version: \u0026#34;20.04\u0026#34; overrides: - name: vmClass value: #@ data.values.cp_vm_size - name: storageClass value: vc01cl01-t0compute replicas: 2 variables: - name: defaultStorageClass value: vc01cl01-t0compute - name: storageClass value: vc01cl01-t0compute - name: storageClasses value: - vc01cl01-t0compute - name: vmClass value: #@ data.values.worker_vm_size - name: ntp value: time1.oc.vmware.com version: v1.23.8+vmware.2-tkg.2-zshippable type: kind: TanzuKubernetesCluster package: vmware.tanzu.manage.v1alpha1.managementcluster.provisioner.tanzukubernetescluster version: v1alpha1 You can see that a number of fields now have YTT logic in them. Here is a quick breakdown of what we are doing.\n#@ load(\u0026quot;@ytt:data\u0026quot;, \u0026quot;data\u0026quot;) - tell ytt to load data values into the data object\n#@ data.values.mgmt_cluster_name - I won\u0026rsquo;t go through every variable but this syntax is used to pull out a value from the values file that we will create in the next section.\n#@ if/end hasattr( data.values, \u0026quot;cp_annotations\u0026quot;): - this syntax is also used a few times, this check to see if our values file has a field and if it does then it adds the field below. This is used becuase certain fields are optional.\nThere is a lot more that can be done when templating with YTT and this is a fairly basic example. The docs on YTT have a lot of example that can be referenced.\nCreate a values file # The values file is what we will use to specify the values for all of the fields that we have templated. This is really just a yaml file with a single line of YTT at the top that let\u0026rsquo;s the YTT engine know that the fields are to be used as data values. Since this is just plain yaml, you can also have nested fields etc. Below you can see the values file that was created to work with the above template.\n#@data/values --- mgmt_cluster_name: h2o-4-19340 cluster_name: cluster-from-template provisioner: lab cp_vm_size: best-effort-large worker_vm_size: best-effort-large cluster_group: default cluster_labels: test: test Create a new cluster # Finally, we can combine these two files into a command that with generate our cluster configuration and then apply it to TMC.\nIf you want to test the outputs of the templating prior to sending it to TMC you can simply run the below command which will generate the resulting yaml and output to stdout.\nytt -f values.yml -f template.yml The next command will generate the resulting yaml and instead of sending it stdout , it will pass it directly to the Tanzu CLI and start creating the cluster.\ntanzu tmc apply -f \u0026lt;(ytt -f values.yml -f template.yml) If you are using EKS or AKS the command is slightly different since support is not yet added to the apply command for those clusters. Hopefully it will be added soon. You can still do this with the create and update commands though. See the examples below.\n# redirecting output does not work currently for the create commands #EKS ytt -f values.yml -f template.yml \u0026gt; eks.yml tanzu tmc ekscluster create -f eks.yml #AKS ytt -f values.yml -f template.yml \u0026gt; aks.yml tanzu tmc akscluster create -f aks.yml Summary # I summary, this article should give you a good idea of how to make re-usable templates for TMC created clusters. This could even be used to create \u0026ldquo;plans\u0026rdquo; for clusters so that it makes self service easier for teams. An example of using this for slef service would be to have a pipeline that executes the apply commands and allow developers, operators, etc. to manage their values file in a git repo. This would provide a nice gitops driven way to create on demand clusters with the ability to apply policy and restrict which fields could be changed.\n","date":"2 November 2023","externalUrl":null,"permalink":"/posts/templating-cluster-creation-with-tanzu-mission-control/","section":"Posts","summary":"Overview # I have had a question come up a few times with customers and coworkers about how to reduce duplication when creating clusters with Tanzu Mission Control(TMC). The question or issue that is usually brought up is that the platform engineering team wants to be able to create clusters quickly and many of the settings between cluster creation are the exact same, thus having a lot of duplication between clusters. When looking at the TMC UI there’s not a way to set custom defaults today to be able to remove the need to fill in every field each time you create a cluster. However, using the UI is probably not the approach a platform team wants to take to scale anyway. It’s much more efficient to codify the clusters and automate the creation. In this post, we will walk through creating cluster templates and using the Tanzu CLI to create clusters with minimal inputs. We will focus on TKG Clusters mostly, but I will also provide some commands that work with AKS and EKS clusters are well.\n","title":"Templating cluster creation with  Tanzu Mission Control","type":"posts"},{"content":"","date":"2 November 2023","externalUrl":null,"permalink":"/tags/tkg/","section":"Tags","summary":"","title":"Tkg","type":"tags"},{"content":"","date":"2 November 2023","externalUrl":null,"permalink":"/tags/tmc/","section":"Tags","summary":"","title":"Tmc","type":"tags"},{"content":"","date":"21 September 2023","externalUrl":null,"permalink":"/tags/antrea/","section":"Tags","summary":"","title":"Antrea","type":"tags"},{"content":"","date":"21 September 2023","externalUrl":null,"permalink":"/tags/nsx/","section":"Tags","summary":"","title":"Nsx","type":"tags"},{"content":"A question comes up often of how can a static IP be set for workloads running in TKG. The answer is generally \u0026ldquo;It depends\u0026rdquo; and then followed by a series of questions about why it\u0026rsquo;s needed and if there are alternatives that could be done etc. In many scenarios, this is needed so that workloads running in a container on TKG can be identified by an external firewall and be allowed to talk to some external service. For example, maybe a workload needs to get access to a particular database and it has a strict access policy based on IP.\nThe overall solution depends on what the full networking stack is that you are using with TKG, but a common challenge is doing this when you are using TKG with supervisor(TKGs) that is backed by NSX-T. The issue is that when using this architecture, all traffic that comes out of a supervisor namespace is routed through a namespace-specific T1 and has a SNAT rule that maps it to a single IP address. This means that all workloads in a supervisor namespace appear as the same IP to the external network.\nIn this article, we will walk through a solution to the problem mentioned above so that we can associate different external IPs with specific workloads running in TKG clusters.\nThe solution # To solve this problem we will use a combination of Antrea and NSX-T. Ultimately we need to be able to make sure that when a container in a pod makes a request, eventually it is associated with a specific IP on the physical network. We also want to potentially have pods in the same cluster that have different external-facing IPs on the network. To do this we can use Antrea egress policies and custom NSX-T SNAT rules.\nAntrea egress policy: This allows us to specify a static IP or pool of IPs that traffic will be SNAT\u0026rsquo;d to when leaving the node if it matches a specific set of labels. You can read all the details here.\nNSX-T SNAT rules: This allows us to specify a rule that matches an IP or range of IPs and translates it to an IP on the physical network as it exits the T0.\nWe can now combine the two features above to solve the problem. The way this works is that the Antrea egress policy will handle making sure that the traffic leaving the node for a specific pod(s) will be a specific IP address, rather than the node\u0026rsquo;s IP address. We can then use the NSX-T SNAT rule to ensure that the IP we specified in the Antrea egress rule is translated to the right external facing IP address. This means that we now have a way to label a pod and have it result in a specific IP address on the physical network.\nThis diagram shows a simplified flow of the IP translations.\nImplementation # This will assume that the following prereqs are met before implementing the solution.\nTKG w/supervisor deployed\nTKG w/supervisor backed my NSX-T\nA TKG workload cluster deployed\nAntrea as a CNI\nCreate the egress policy # The first step is to create the Antrea egress policy and IP pool. This will be where the label matching is set so that we can target specific pods or namespaces in the cluster.\ncreate the following YAML file and apply it to your workload cluster.\napiVersion: crd.antrea.io/v1alpha2 kind: Egress metadata: name: egress-external-db spec: appliedTo: podSelector: matchLabels: external-db-access: \u0026#39;true\u0026#39; externalIPPool: external-ip-pool --- apiVersion: crd.antrea.io/v1alpha2 kind: ExternalIPPool metadata: name: external-ip-pool spec: ipRanges: - start: 10.244.0.70 # IP from segment that nodes are on end: 10.244.0.70 nodeSelector: {} # All Nodes can be Egress Nodes After applying the file you should see a status like this:\nk get egresses.crd.antrea.io egress-external-db NAME EGRESSIP AGE NODE egress-external-db 10.244.0.70 18s tes-snat-1-md-0-bzfm4-7678dc8766-w7rz2 In the above file, you can see that we are creating two resources. the Egress resource defines the match criteria and the IP pool we will use. The ExternalIPPool defines the IP range that we want to use. In this case, it\u0026rsquo;s set to just 1 IP. This IP is pulled from the cluster\u0026rsquo;s segment CIDR in NSX-T, so it is an IP that would typically be used for a node. This IP could be from another range but for simplicity, we are using the segment\u0026rsquo;s CIDR.\nCreate a workload with matching labels # We now need to create a workload that matches our egress policy label selectors. This will start up a pod that matches the labels, in this example, we are using netshoot which is an open-source tool for network troubleshooting.\nApply the following YAML into the workload cluster.\napiVersion: apps/v1 kind: Deployment metadata: name: netshoot labels: app: netshoot spec: replicas: 1 selector: matchLabels: app: netshoot template: metadata: labels: app: netshoot external-db-access: \u0026#39;true\u0026#39; spec: containers: - name: netshoot image: nicolaka/netshoot command: [\u0026#34;/bin/bash\u0026#34;] args: [\u0026#34;-c\u0026#34;, \u0026#34;while true; do ping localhost; sleep 60;done\u0026#34;] At this point, the traffic leaving this pod will be translated to our Antrea egress IP 10.244.0.70 . However we are not finished just yet, the IP at this point will still be translated to the default NSX-T egress IP when leaving the T0 because our default SNAT rule that is created by TKG is capturing all traffic on that segment\u0026rsquo;s subnet.\nAdd the custom NSX-T SNAT rule # This is where we will create a rule in NSX-T to translate the Antrea egress IP to the final outbound IP address. This can be done either through the UI or automation with the API, etc. For simplicity, this shows how to create the rule in the UI.\ngo into the NSX-T console and navigate to the Networking-\u0026gt;NAT section.\nfind the T1 that is associated with your supervisor namespace in the dropdown. the T1 name will contain the ns name.\ncreate a new SNAT rule, see the image below for details. This will set the Antrea egress IP as the source and the destination is the outbound IP we want from the NSX-T egress range.\nWe can now test to see if the IP is being translated properly.\nValidate it # To validate this we will exec into the container and ping a Linux box running on a different network while watching traffic with tcpdump. This will make the traffic flow the full course of the network and we should see the outbound translated IP.\nRun the ping command to the Linux box on 10.220.13.144\nk exec -it netshoot-75dd7f67c9-zck8f -- bash netshoot-75dd7f67c9-zck8f: ping netshoot-75dd7f67c9-zck8f: ping 10.220.13.144 PING 10.220.13.144 (10.220.13.144) 56(84) bytes of data. Check the tcpdump on the Linux box. Notice the IP is 10.214.185.200 which is our custom SNAT rule IP.\ntcpdump -i eth0 icmp tcpdump: verbose output suppressed, use -v[v]... for full protocol decode listening on eth0, link-type EN10MB (Ethernet), snapshot length 262144 bytes 17:16:23.009148 IP 10.214.185.200 \u0026gt; photon-machine: ICMP echo request, id 2823, seq 1, length 64 Summary # In summary, using this approach will allow you to be able to conform to existing firewall rules and policies that rely on specifying an IP address for specific workloads. This could be used for many different use cases that require this type of granularity. additionally it lets us use a labeling and code-based mechanism to assign these IPs to workloads and can be very dynamic if needed.\n","date":"21 September 2023","externalUrl":null,"permalink":"/posts/setting-static-ips-for-workloads-in-tkg/","section":"Posts","summary":"A question comes up often of how can a static IP be set for workloads running in TKG. The answer is generally “It depends” and then followed by a series of questions about why it’s needed and if there are alternatives that could be done etc. In many scenarios, this is needed so that workloads running in a container on TKG can be identified by an external firewall and be allowed to talk to some external service. For example, maybe a workload needs to get access to a particular database and it has a strict access policy based on IP.\n","title":"Setting static IPs for workloads in TKG","type":"posts"},{"content":"I was working with a customer the other day and some questions came up about how to support workloads that may vary from the ones that TAP supports out-of-the-box. Looking at the docs we can see that there are 3 types available to use today. This will support the majority of workloads that are commonly used, but what if another use case comes up and we need to quickly add support for deploying that workload on our existing TAP environment? This post will walk through the basic steps of adding a new workload type into TAP.\nHow it works # First, let\u0026rsquo;s break down what is meant by a \u0026ldquo;custom workload type\u0026rdquo;. In TAP there is a resource called a \u0026ldquo;workload\u0026rdquo; which defines some minimal details about an application and abstracts away many of the underlying complexities of the infrastructure and the path that the app will take to get to a running state on that infrastructure i.e. the path to prod. The \u0026ldquo;workload\u0026rdquo; definition makes it easy for a developer to supply some basic app configuration and source code and get the app running on K8s with little to no knowledge of the underlying K8s environment. In the workload definition, there is a label that is used to set the type of workload that will be created. Based on that label\u0026rsquo;s value, an instance of the supply chain is created for the workload and will generate a set of K8s resources that are determined by the type. For example the server type will generate a K8s Deployment and Service , while if the web type is specified it will generate a Knative Service. By creating a custom workload type we can define which K8s resources get generated when that type is specified.\nTAP provides a way to include new workload types via a setting in the values.yaml called ootb_supply_chain_basic.supported_workloads, more on this in the implementation section below. What\u0026rsquo;s great about this is that this makes it easy to add a custom workload type without having to modify any supply chains or do any real \u0026ldquo;customization\u0026rdquo; that strays from the OOTB offerings. In addition to the TAP values file modification, a new ClusterConfigurationTemplate needs to be added. The ClusterConfigurationTemplate is where the K8s resources that will be generated for the type are defined.\nHere is a high-level outline of the end-to-end process:\nA workload is defined with the custom workload type as the value for the label.\nThe workload is applied to the build cluster which instantiates an instance of the supply chain.\nWhen the supply chain instance is created there is a selection that happens on the app-config step that looks at the workload type label and chooses the correct ClusterConfigurationTemplate , which in this case would be the custom one.\nThe supply chain eventually reaches the app-config step and the custom template stamps out a ConfigMap with the custom-defined K8s resources that will be eventually deployed on the TAP run clusters.\nDiagram of this flow:\nThe use case # As mentioned in the introduction we will be creating a new workload type for a use case that isn\u0026rsquo;t currently covered by the out-of-the-box types. For this use case, we have an app that needs to mount a volume. Specifically, this app needs to mount a K8s persistent volume to store some data.\nImplementation # Create a new Cluster Configuration Template # The ClusterConfigurationTemplate resource is used to define the resulting delivery YAML. This YAML will consist of the Kubernetes manifests that are responsible for running the application.\nApply the below YAML into the TAP build cluster.\napiVersion: carto.run/v1alpha1 kind: ClusterConfigTemplate metadata: annotations: doc: | This template consumes an input named config which contains a PodTemplateSpec and returns a ConfigMap which contains a \u0026#34;delivery.yml\u0026#34; which contains a manifests for a Kubernetes Deployment which will run the templated pod, and a \u0026#34;service.yml\u0026#34; Kubernetes Service to expose the pods on the network. This also supports a workload param that allows for adding volumes name: volumes-template spec: configPath: .data healthRule: alwaysHealthy: {} params: - default: - containerPort: 8080 name: http port: 8080 name: ports ytt: | #@ load(\u0026#34;@ytt:data\u0026#34;, \u0026#34;data\u0026#34;) #@ load(\u0026#34;@ytt:yaml\u0026#34;, \u0026#34;yaml\u0026#34;) #@ load(\u0026#34;@ytt:struct\u0026#34;,\u0026#34;struct\u0026#34;) #@ load(\u0026#34;@ytt:assert\u0026#34;, \u0026#34;assert\u0026#34;) #@ load(\u0026#34;@ytt:overlay\u0026#34;, \u0026#34;overlay\u0026#34;) #@ def merge_labels(fixed_values): #@ labels = {} #@ if hasattr(data.values.workload.metadata, \u0026#34;labels\u0026#34;): #@ labels.update(data.values.workload.metadata.labels) #@ end #@ labels.update(fixed_values) #@ return labels #@ end #@ def intOrString(v): #@ return v if type(v) == \u0026#34;int\u0026#34; else int(v.strip()) if v.strip().isdigit() else v #@ end #@ def merge_ports(ports_spec,containers): #@ ports = {} #@ for c in containers: #@ for p in getattr(c,\u0026#34;ports\u0026#34;, []): #@ ports[p.containerPort] = {\u0026#34;targetPort\u0026#34;: p.containerPort,\u0026#34;port\u0026#34;: p.containerPort, \u0026#34;name\u0026#34;: getattr(p, \u0026#34;name\u0026#34;, str(p.containerPort))} #@ end #@ end #@ for p in ports_spec: #@ targetPort = getattr(p,\u0026#34;containerPort\u0026#34;, p.port) #@ type(targetPort) in (\u0026#34;string\u0026#34;, \u0026#34;int\u0026#34;) or fail(\u0026#34;containerPort must be a string or int\u0026#34;) #@ targetPort = intOrString(targetPort) #@ #@ port = p.port #@ type(port) in (\u0026#34;string\u0026#34;, \u0026#34;int\u0026#34;) or fail(\u0026#34;port must be a string or int\u0026#34;) #@ port = int(port) #@ ports[p.port] = {\u0026#34;targetPort\u0026#34;: targetPort, \u0026#34;port\u0026#34;: port, \u0026#34;name\u0026#34;: getattr(p, \u0026#34;name\u0026#34;, str(p.port))} #@ end #@ return ports.values() #@ end #@ def addVolumes(): spec: containers: #@overlay/match by=\u0026#34;name\u0026#34; - name: workload #@overlay/match missing_ok=True volumeMounts: #@ data.values.params.volumes.volumeMounts #@overlay/match missing_ok=True volumes: #@ data.values.params.volumes.volumes #@ end #@ def delivery(): --- apiVersion: apps/v1 kind: Deployment metadata: name: #@ data.values.workload.metadata.name annotations: kapp.k14s.io/update-strategy: \u0026#34;fallback-on-replace\u0026#34; ootb.apps.tanzu.vmware.com/servicebinding-workload: \u0026#34;true\u0026#34; labels: #@ merge_labels({ \u0026#34;app.kubernetes.io/component\u0026#34;: \u0026#34;run\u0026#34;,\u0026#34;carto.run/workload-name\u0026#34;: data.values.workload.metadata.name }) spec: selector: matchLabels: #@ data.values.config.metadata.labels template: #@ overlay.apply(data.values.config,addVolumes()) --- apiVersion: v1 kind: Service metadata: name: #@ data.values.workload.metadata.name labels: #@ merge_labels({ \u0026#34;app.kubernetes.io/component\u0026#34;: \u0026#34;run\u0026#34;, \u0026#34;carto.run/workload-name\u0026#34;: data.values.workload.metadata.name }) spec: selector: #@ data.values.config.metadata.labels ports: #@ hasattr(data.values.params, \u0026#34;ports\u0026#34;) and len(data.values.params.ports) or assert.fail(\u0026#34;one or more ports param must be provided.\u0026#34;) #@ declared_ports = {} #@ if \u0026#34;ports\u0026#34; in data.values.params: #@ declared_ports = data.values.params.ports #@ else: #@ declared_ports = struct.encode([{ \u0026#34;containerPort\u0026#34;: 8080, \u0026#34;port\u0026#34;: 8080, \u0026#34;name\u0026#34;: \u0026#34;http\u0026#34;}]) #@ end #@ for p in merge_ports(declared_ports, data.values.config.spec.containers): - #@ p #@ end #@ end --- apiVersion: v1 kind: ConfigMap metadata: name: #@ data.values.workload.metadata.name + \u0026#34;-server\u0026#34; labels: #@ merge_labels({ \u0026#34;app.kubernetes.io/component\u0026#34;: \u0026#34;config\u0026#34;}) data: delivery.yml: #@ yaml.encode(delivery()) Let\u0026rsquo;s break this template down a bit.\nThe resource takes as input a PodTemplateSpec from the previous step in the supply chain. This is accessible through the data.values.config .\nThe resource defines a ytt template that will be used to generate a ConfigMap that holds the delivery.yaml . The delivery.yaml contains the Kubernetes manifests to run the app.\nThe delivery() function - Defines a templated K8s deployment and service using input from the workload params and the previous step\u0026rsquo;s PodTemplateSpec .\nThe addVolumes() function - this is used within the delivery function to modify the incoming PodTemplateSpec , it will take the volume params provided in the workload spec and add them to the PodTemplateSpec . This is the part that is doing most of the custom work to enable volumes.\nThe merge_ports() function - this takes in any custom ports defined in the params in the workload spec and adds them to the service.\nThe merge_labels() function - this does a similar task to the merge_ports but this time with any labels passed in.\nUpdate the TAP values to include the new workload type # To register this new workload type a new section needs to be added to the supply chain\u0026rsquo;s configuration.\nEdit the TAP values and add the below YAML. Notice this is being added to the ootb_supply_chain_basic section so just append this along with any other settings. After updating these settings, update the tap install using the Tanzu cli.\nootb_supply_chain_basic: supported_workloads: - type: web cluster_config_template_name: config-template - type: server cluster_config_template_name: server-template - type: worker cluster_config_template_name: worker-template - type: server-with-volumes cluster_config_template_name: volumes-template The settings above define four workload types, three of which are provided OOTB. The existing three need to be specified otherwise they will be removed. The fourth is the custom workload type that has been named server-with-volumes and is referencing the new ClusterConfigurationTemplate named volumes-template . These settings are what associate the workload type that will be used in the workload spec with the new template defined in the previous step.\nCreate a workload using the new type # Keeping with the thread of our new use case, for this workload we need to mount a persistent volume. Creating the volume is out of the scope of this post, but any K8s PVC will work. In this case, the PVC is just using the AWS EBS CSI driver to create a disk.\nApply the below YAML into the TAP build cluster\napiVersion: carto.run/v1alpha1 kind: Workload metadata: labels: app.kubernetes.io/part-of: go-sample-pvc apps.tanzu.vmware.com/workload-type: server-with-volumes name: go-sample-pvc namespace: default spec: params: - name: volumes value: volumes: - name: data-mount persistentVolumeClaim: claimName: go-sample-data volumeMounts: - name: data-mount mountPath: /sample-data source: git: ref: branch: main url: https://github.com/warroyo/tap-go-sample The main things to point out in the above YAML are the following:\napps.tanzu.vmware.com/workload-type: server-with-volumes this label is what is used for selecting the workload type. Notice that the server-with-volumes name matches the one we added to the TAP values.\nThe volumes param - This is where the volumes and volume mounts are defined. To keep it simple and flexible the format of these parameters is just using the same spec from the K8s pod spec for defining volumes. This means anything that can be done through these docs, can be added here.\nAfter deploying this workload the resulting K8s manifests will look like this. Notice that the volume and volume mounts have been added to the pod spec.\napiVersion: apps/v1 kind: Deployment metadata: name: go-sample-pvc annotations: null kapp.k14s.io/update-strategy: fallback-on-replace ootb.apps.tanzu.vmware.com/servicebinding-workload: \u0026#34;true\u0026#34; labels: app.kubernetes.io/part-of: go-sample-pvc apps.tanzu.vmware.com/has-tests: \u0026#34;true\u0026#34; apps.tanzu.vmware.com/workload-type: server-with-volumes app.kubernetes.io/component: run carto.run/workload-name: go-sample-pvc spec: selector: matchLabels: app.kubernetes.io/component: run app.kubernetes.io/part-of: go-sample-pvc apps.tanzu.vmware.com/has-tests: \u0026#34;true\u0026#34; apps.tanzu.vmware.com/workload-type: server-with-volumes carto.run/workload-name: go-sample-pvc template: metadata: annotations: conventions.carto.run/applied-conventions: |- spring-boot-convention/auto-configure-actuators-check spring-boot-convention/app-live-view-appflavour-check appliveview-sample/app-live-view-appflavour-check developer.conventions/target-containers: workload labels: app.kubernetes.io/component: run app.kubernetes.io/part-of: go-sample-pvc apps.tanzu.vmware.com/has-tests: \u0026#34;true\u0026#34; apps.tanzu.vmware.com/workload-type: server-with-volumes carto.run/workload-name: go-sample-pvc spec: containers: - image: dev.registry.pivotal.io/warroyo/go-sample-pvc-default@sha256:94eec8cb112d4e860ab6cc9095f40db0779889f7ee0a953f0876d4b4b29ef0ce name: workload resources: {} securityContext: runAsUser: 1000 volumeMounts: - mountPath: /sample-data name: data-mount serviceAccountName: default volumes: - name: data-mount persistentVolumeClaim: claimName: go-sample-data --- apiVersion: v1 kind: Service metadata: name: go-sample-pvc labels: app.kubernetes.io/part-of: go-sample-pvc apps.tanzu.vmware.com/has-tests: \u0026#34;true\u0026#34; apps.tanzu.vmware.com/workload-type: server-with-volumes app.kubernetes.io/component: run carto.run/workload-name: go-sample-pvc spec: selector: app.kubernetes.io/component: run app.kubernetes.io/part-of: go-sample-pvc apps.tanzu.vmware.com/has-tests: \u0026#34;true\u0026#34; apps.tanzu.vmware.com/workload-type: server-with-volumes carto.run/workload-name: go-sample-pvc ports: - targetPort: 8080 port: 8080 name: http Summary # In summary, the steps in this post walked through adding a new configuration template that takes parameters from the workload spec and uses those to inject volumes into the resulting pods. We then associated that new configuration template with a new workload type that can be used by a developer when creating a workload. This approach could be used for many other use cases and is not limited to adding volumes. Hopefully, this shows how TAP can be extended to support all kinds of different workloads and application needs while also providing a good user experience for the developer.\n","date":"19 January 2023","externalUrl":null,"permalink":"/posts/how-to-create-custom-workload-types-with-tap/","section":"Posts","summary":"I was working with a customer the other day and some questions came up about how to support workloads that may vary from the ones that TAP supports out-of-the-box. Looking at the docs we can see that there are 3 types available to use today. This will support the majority of workloads that are commonly used, but what if another use case comes up and we need to quickly add support for deploying that workload on our existing TAP environment? This post will walk through the basic steps of adding a new workload type into TAP.\n","title":"How to create custom workload types with TAP","type":"posts"},{"content":"","date":"28 December 2022","externalUrl":null,"permalink":"/tags/azure/","section":"Tags","summary":"","title":"Azure","type":"tags"},{"content":"","date":"28 December 2022","externalUrl":null,"permalink":"/tags/azure-devops/","section":"Tags","summary":"","title":"Azure-Devops","type":"tags"},{"content":"TAP has an OOTB source code testing capability that makes use of Tekton pipelines to execute tests based on your workload types. However, many organizations have already implemented their testing processes in another tool like Jenkins or Azure DevOps (ADO). As of TAP 1.3, you can natively use Jenkins for your source testing in the TAP supply chain. In talking with several customers and co-workers it was apparent that integrating with ADO would be very useful. In this post, we will walk through the steps to get the TAP source code testing capability working with Azure DevOps pipelines.\nHow it works # Since TAP already has an integration with Jenkins, we will follow a similar pattern for implementing the ADO integration. Below is a breakdown of how source testing works in TAP with Jenkins. The diagram shows that there is a parameter for testing_pipeline_matching_labels in the workload definition, this is what is used to select the pipeline, via label selectors, that should be used for testing. From there a Runnable is created and uses the label selectors to associate the Jenkins Tekton pipeline. There is also a ClusterTask that the pipeline references; this is where the code exists for communicating with Jenkins. From there, when the supply chain executes it will create a PipelineRun with the parameters from the workload that will spawn a TaskRun.The task run executes the Jenkins job and returns the results. There are a few pieces I left out of the diagram to make it easier to follow but, overall this covers most of what happens.\nBased on the above flow, there will be two things needed to implement an ADO equivalent: a custom ClusterTask and a Pipeline. With those defined, we will be able to use native TAP functionality to selectively run tests in ADO. These two resources are core components of Tekton, you can find the docs on them here.\nImplementation # Create a simple pipeline in ADO # Login to ADO and create a \u0026ldquo;new project\u0026rdquo; or use an existing project that you may have. If you are using the Azure CLI, run the below command.\naz devops project create --name tap-ado-blog --org https://dev.azure.com/\u0026lt;your-organization\u0026gt; After creating the new project there will also be a default repo created by the same name, e.g. tap-ado-blog. This is the repo that will be used for the pipeline in the next step. You can also create a new repo or use an existing one, just be sure to change the names in the next steps accordingly.\nThe below YAML will be used for the newly created pipeline. This sets up three parameters, two of which are required since the TAP supply chain passes the source_url and source_revision by default. The third is an optional parameter that shows how to pass additional parameters to the ADO pipeline in the workload YAML. The pipeline steps can be customized to handle any testing scenario needed.\ntrigger: - none pr: none pool: vmImage: ubuntu-latest parameters: - name: source_url displayName: source url to clone type: string - name: source_revision displayName: revision to clone type: string - name: example_param displayName: example default: \u0026#34;\u0026#34; type: string steps: - script: echo ${{parameters.source_url}} \u0026#34;succesfully triggered this build from TAP\u0026#34; - script: echo ${{parameters.source_revision}} \u0026#34;succesfully triggered this build from TAP\u0026#34; Next, create a new pipeline\u0026hellip;\nFrom the UI: make the following selections Pipelines-\u0026gt;Create pipeline-\u0026gt;Azure Repos Git-\u0026gt;tap-ado-blog-\u0026gt;Starter Pipeline. Add the above YAML after selecting the \u0026ldquo;Starter Pipeline\u0026rdquo; and save it.\nUsing the CLI: commit the above YAML to the newly created repo as azure-pipelines.yml , run the below commands.\ngit clone https://\u0026lt;your-org\u0026gt;@dev.azure.com/\u0026lt;your-org\u0026gt;/tap-ado-blog/_git/tap-ado-blog cd tap-ado-blog touch azure-pipelines.yml #paste the contents from the above yaml into the new file git add . git commit -am \u0026#34;adding pipelines\u0026#34; git push az pipelines create --name \u0026#39;tap-ado-blog\u0026#39; --description \u0026#39;Pipeline for TAP\u0026#39; --repository tap-ado-blog --branch main --repository-type tfsgit --org https://dev.azure.com/\u0026lt;your-organization\u0026gt; --project tap-ado-blog --yaml-path azure-pipelines.yml Create a PAT in Azure DevOps # To execute the pipeline from TAP there needs to be a PAT created to auth against the API. If you are running TAP in Azure you could do something like role-based access control to provide auth but that is for another blog post.\nFrom the UI: select Upper right settings-\u0026gt;Personal Access Tokens-\u0026gt;New Token-\u0026gt;Full Access , see here for the full docs.\nUsing the CLI: as of 12/28/2022, there is no way to get a PAT from the CLI except for going direct to the REST API.\nSave the generated PAT for later use.\nRelocate the required image to your registry # In the docs for TAP it will typically have you relocate your images to a registry during the installation. The new ClusterTask that will be created for this ADO integration will require a new image due to its Python dependency. Run the following command after exporting the required variables to relocate the image to your registry.\nimgpkg copy -i python:3.7-slim --to-repo ${INSTALL_REGISTRY_HOSTNAME}/${INSTALL_REPO}/tap-packages After running the command you will see some output similar to\nwill export index.docker.io/library/python@sha256:aa949f5f10e9b28e1f9561fff73d1a359fa8517d4e543451a714d1a4ecc61c56 To get the full path to the copied image, copy everything starting with the @ symbol in your output and append it to the new repo path. The resulting full image path will look something like\n${INSTALL_REGISTRY_HOSTNAME}/${INSTALL_REPO}/tap-packages@sha256:aa949f5f10e9b28e1f9561fff73d1a359fa8517d4e543451a714d1a4ecc61c56 # for example: https://dev.registry.pivotal.io/warroyo/tap-packages@sha256:aa949f5f10e9b28e1f9561fff73d1a359fa8517d4e543451a714d1a4ecc61c56 Depending on when you run this the SHA may be different, but the same process applies to get the full path to the image.\nCreate the TAP resources # NOTE: The next few steps will walk you through creating the required TAP resources. This should be done in the \u0026ldquo;build\u0026rdquo; cluster, or if you are using a \u0026ldquo;full profile\u0026rdquo;, it will be in the single cluster since the build components are colocated. Ensure you are in the correct context when running these commands.\nCreate a Kubernetes secret to store the PAT from the previous section. This should be done in the \u0026ldquo;developer namespace\u0026rdquo;.\nkubectl -n \u0026lt;developer-ns\u0026gt; create secret generic ado-token --from-literal=pat=\u0026lt;your-pat-here\u0026gt; Create a new ClusterTask, this will set up the code that will be executed to run the pipeline in Azure. In the Below YAML, replace \u0026lt;your-relocated-image-here\u0026gt; with the image from the above step. After modifying, apply this to the cluster with kubectl.\napiVersion: tekton.dev/v1beta1 kind: ClusterTask metadata: name: ado-task spec: params: - name: source-url type: string - name: source-revision type: string - name: secret-name type: string - name: pipeline-id type: string - name: project-name type: string - name: org-name type: string - default: \u0026#34;\u0026#34; name: pipeline-params type: string results: - name: ado-pipeline-run-url type: string steps: - name: install-depends image: \u0026lt;your-relocated-image-here\u0026gt; script: | pip install requests - env: - name: ADO_API_TOKEN valueFrom: secretKeyRef: key: pat name: $(params.secret-name) - name: SOURCE_URL value: $(params.source-url) - name: PIPELINE_PARAMS value: $(params.pipeline-params) - name: SOURCE_REVISION value: $(params.source-revision) - name: PIPELINE_ID value: $(params.pipeline-id) - name: ORG_NAME value: $(params.org-name) - name: PROJECT_NAME value: $(params.project-name) image: \u0026lt;your-relocated-image-here\u0026gt; name: trigger-ado-build script: | #!/usr/bin/env bash set -o errexit set -o pipefail pip install requests python3 \u0026lt;\u0026lt; END import os import subprocess import logging import sys import time import json import requests logging.basicConfig(level=logging.DEBUG) org = os.getenv(\u0026#39;ORG_NAME\u0026#39;) project = os.getenv(\u0026#39;PROJECT_NAME\u0026#39;) pipeline = os.getenv(\u0026#39;PIPELINE_ID\u0026#39;) token = os.getenv(\u0026#39;ADO_API_TOKEN\u0026#39;) source_url = os.getenv(\u0026#39;SOURCE_URL\u0026#39;) source_revision = os.getenv(\u0026#39;SOURCE_REVISION\u0026#39;) pipeline_params = os.getenv(\u0026#39;PIPELINE_PARAMS\u0026#39;) url = f\u0026#39;https://dev.azure.com/{org}/{project}/_apis/pipelines/{pipeline}/runs?api-version=7.0\u0026#39; existing_params = { \u0026#34;source_url\u0026#34;: f\u0026#39;{source_url}\u0026#39;, \u0026#34;source_revision\u0026#34;: f\u0026#39;{source_revision}\u0026#39; } input_params = {} if pipeline_params != \u0026#34;\u0026#34;: input_params = json.loads(pipeline_params) existing_params.update(input_params) payload = json.dumps({ \u0026#34;templateParameters\u0026#34;: existing_params }) headers = { \u0026#39;Content-Type\u0026#39;: \u0026#39;application/json\u0026#39; } pipelineResponse = requests.request(\u0026#34;POST\u0026#34;, url, headers=headers, data=payload,auth=(\u0026#39;\u0026#39;,token)) logging.info(pipelineResponse.text) #throw error if not 200 pipelineResponse.raise_for_status() #check status of pipeline run and validate it succeeds jsonResponse = pipelineResponse.json() currentRun = jsonResponse[\u0026#39;_links\u0026#39;][\u0026#39;self\u0026#39;][\u0026#39;href\u0026#39;] results_url = jsonResponse[\u0026#39;_links\u0026#39;][\u0026#39;web\u0026#39;][\u0026#39;href\u0026#39;] f = open(\u0026#34;$(results.ado-pipeline-run-url.path)\u0026#34;, \u0026#34;w\u0026#34;) f.write(results_url) f.close() running = True while running: response = requests.get(currentRun, headers=headers, auth=(\u0026#39;\u0026#39;,token), timeout=300) response.raise_for_status() result = response.json() if result[\u0026#39;state\u0026#39;] != \u0026#39;completed\u0026#39;: logging.info(f\u0026#34;pipeline state is {result[\u0026#39;state\u0026#39;]}, entering sleep for 5 seconds\u0026#34;) time.sleep(5) elif result[\u0026#39;result\u0026#39;] == \u0026#39;succeeded\u0026#39;: logging.info(f\u0026#34;pipeline was successful, exiting\u0026#34;) sys.exit(os.EX_OK) else: logging.info(f\u0026#34;pipeline result is {result[\u0026#39;result\u0026#39;]}, check ADO\u0026#34;) sys.exit(os.EX_SOFTWARE) END As you will notice, the main portion of the above file is a script. I chose Python for this since it was much easier to read than the Bash equivalent. Taking a deeper look into what is happening in the script we see the following:\nparameters are being passed in from the supply chain\na REST call is made to ADO to execute the pipeline\na while loop is used to continuously execute a REST call to ADO until the status is succeeded or failed\nNext create a new Pipeline that will reference the ClusterTask and have the required labels so that it can be selected by the supply chain. This should also be created in the \u0026ldquo;developer namespace\u0026rdquo;.\n--- apiVersion: tekton.dev/v1beta1 kind: Pipeline metadata: name: developer-defined-ado-pipeline namespace: \u0026lt;developer-ns\u0026gt; labels: #! This label should be provided to the Workload so that #! the supply chain can find this pipeline apps.tanzu.vmware.com/pipeline: ado-pipeline spec: results: - name: ado-pipeline-run-url #! To show the job URL on the #! Tanzu Application Platform GUI value: $(tasks.ado-task.results.ado-pipeline-run-url) params: - name: source-url #! Required - name: source-revision #! Required - name: secret-name #! Required - name: project-name #! Required - name: pipeline-id #! Required - name: org-name #! Required - name: pipeline-params default: \u0026#34;\u0026#34; tasks: - name: ado-task taskRef: name: ado-task kind: ClusterTask params: - name: source-url value: $(params.source-url) - name: source-revision value: $(params.source-revision) - name: secret-name value: $(params.secret-name) - name: pipeline-id value: $(params.pipeline-id) - name: org-name value: $(params.org-name) - name: project-name value: $(params.project-name) - name: pipeline-params value: $(params.pipeline-params) Finally, create or modify a workload to use the new labels and parameters that will trigger the ADO testing pipeline. The below workload can be used since the GitHub repo is public and can be cloned without credentials. Just replace the org-name and pipeline-id with the ones from your account. The pipeline-id can be found in the URL when looking at the pipeline in the browser.\n--- apiVersion: carto.run/v1alpha1 kind: Workload metadata: labels: app.kubernetes.io/part-of: company-api-ado apps.tanzu.vmware.com/has-tests: \u0026#34;true\u0026#34; apps.tanzu.vmware.com/workload-type: web name: company-api-ado namespace: default spec: build: env: - name: BP_KEEP_FILES value: \u0026#34;docs/*\u0026#34; source: git: ref: branch: main url: https://github.com/warroyo/tap-go-sample params: - name: testing_pipeline_matching_labels value: apps.tanzu.vmware.com/pipeline: ado-pipeline - name: testing_pipeline_params value: project-name: tap-ado-blog secret-name: ado-token org-name: \u0026lt;your-org\u0026gt; pipeline-id: \u0026lt;pipeline-id\u0026gt; pipeline-params: \u0026#34;{\\\u0026#34;newparam\\\u0026#34;: \\\u0026#34;testing\\\u0026#34;}\u0026#34; Once this workload is created the source-tester step should now execute the pipeline in ADO.\nFinal Result # The final result is illustrated in the diagram below. The workload creates a supply chain and as part of that the source-tester step is executed. The source-tester triggers the ADO pipeline defined in the workload and will either fail and stop the supply chain or succeeded and continue the supply chain.\nConclusion # In summary, the steps in this blog added an option to our TAP supply chain to use ADO pipelines for testing code. One really nice thing about this is that it didn\u0026rsquo;t require any modifications to the supply chains or any core TAP components. Through the native label selection that TAP provides we were able to add another testing option in addition to the OOTB options. This also does not need to be limited to testing code in the ADO pipeline. In a future post, I will cover how the same approach could be used to add steps to the supply chain that will run some arbitrary steps in an ADO Pipeline. This should give an idea of the possibilities that exist for integrating TAP with other toolsets.\n","date":"28 December 2022","externalUrl":null,"permalink":"/posts/integrating-tap-with-ado-pipelines/","section":"Posts","summary":"TAP has an OOTB source code testing capability that makes use of Tekton pipelines to execute tests based on your workload types. However, many organizations have already implemented their testing processes in another tool like Jenkins or Azure DevOps (ADO). As of TAP 1.3, you can natively use Jenkins for your source testing in the TAP supply chain. In talking with several customers and co-workers it was apparent that integrating with ADO would be very useful. In this post, we will walk through the steps to get the TAP source code testing capability working with Azure DevOps pipelines.\n","title":"Integrating  TAP with Azure DevOps Pipelines","type":"posts"},{"content":"Hey! I am Will Arroyo.\nI am a Solution Engineer at VMware living in Denver and working on all things Cloud Native with a heavy focus on Kubernetes.\nIn the past, I have helped build large-scale infrastructure automation systems as well as helped modernize software deployments across enterprises.\nI am also passionate about cooking and food in general so you may see some posts about food mixed into this blog.\nYou can find me on GitHub.\nThis blog runs on Hugo with the Blowfish theme and is hosted on Cloudflare Pages.\n","externalUrl":null,"permalink":"/about/","section":"Will Arroyo","summary":"Hey! I am Will Arroyo.\nI am a Solution Engineer at VMware living in Denver and working on all things Cloud Native with a heavy focus on Kubernetes.\nIn the past, I have helped build large-scale infrastructure automation systems as well as helped modernize software deployments across enterprises.\n","title":"About","type":"page"},{"content":"","externalUrl":null,"permalink":"/authors/","section":"Authors","summary":"","title":"Authors","type":"authors"},{"content":"","externalUrl":null,"permalink":"/categories/","section":"Categories","summary":"","title":"Categories","type":"categories"},{"content":"","externalUrl":null,"permalink":"/series/","section":"Series","summary":"","title":"Series","type":"series"}]