Provisioning a VKS cluster is the easy part. Keeping it alive six months later, with upgrades, capacity management, and incidents that don’t wait, is what separates a real platform from a POC dressed as production.
This article walks through the day-2 operations that frame a seriously-operated VKS cluster: CAPI lifecycle, upgrades, backup, observability, and scaling. The goal is to give reference practices and the traps the official docs don’t highlight.
Audience: you already have a running VKS cluster (see the architect’s guide) and you’re preparing for production.
About this article
Synthesis based on Broadcom VCF 9 documentation, upstream Cluster API patterns, and community feedback around VKr upgrades. The value is in the sequence of steps and the orderings to respect.
CAPI lifecycle in practice
VKS is built on Cluster API. Understanding its primitives is understanding how you operate your cluster — every action goes through a YAML manifest applied to the Supervisor.
Cluster — the root resource. Describes the desired topology (control plane replicas, workers, version). Editing its fields triggers reconciliation by CAPI.
MachineDeployment — the equivalent of a Kubernetes Deployment but for node VMs. Manages a worker pool with native rolling updates.
Machine — an individual VM. Has its own lifecycle (provisioning, running, deleting). Deleting a Machine triggers replacement by CAPI.
KubeadmControlPlane — manages the control plane. Encapsulates the etcd + API server upgrade logic, which doesn’t tolerate approximation.
ClusterClass — the parameterizable template installed by the Supervisor. It’s what turns a standard CAPI manifest into a VKS-specific cluster.
The rule that changes everything. All modifications go through the manifest. No kubectl edit directly on Machines. No manual edits of VMs in vCenter. If you need to “fix” a node, you delete it through CAPI and let reconciliation recreate it. It’s counterintuitive coming from classic vSphere, but it’s the only way to keep state coherent.
Scaling: horizontal vs vertical
Two dimensions to scale, two different mechanics.
Horizontal (adding nodes). Edit replicas in the worker pool’s MachineDeployment. Going from N to N+1 triggers provisioning of a new worker. Going down triggers a clean drain + delete — pods are evicted according to their PDBs. This is the routine, safe operation.
Vertical (changing node sizing). Editing the vmClass (e.g. guaranteed-medium → guaranteed-large) triggers a rolling replace: each existing node is progressively replaced by a node of the new class. Heavier, and during the operation the cluster temporarily holds more nodes than the target. Plan namespace quotas to absorb that transient surplus.
Cluster Autoscaler: not enabled by default
VKS does not enable Cluster Autoscaler by default. You need to deploy and configure it explicitly to interact with CAPI. Without it, the cluster doesn’t grow on its own even with Pending pods. In most cases that’s deliberate — an autoscaler enabled without governance can blow up your namespace quota cost. Enable it once operational maturity is sufficient.
Kubernetes upgrades: the sequence to respect
The Supervisor / VKS version decoupling is the platform’s main operational argument. You still have to use it correctly.
┌─────────────────────────────────────────────────────────┐
│ Supervisor (vCenter embed) │
│ ├── VKr v1.30 (cluster A) ← upgradable │
│ ├── VKr v1.31 (cluster B) ← independently │
│ └── VKr v1.32 (cluster C) ← of one another │
└─────────────────────────────────────────────────────────┘Step 1 — Fetch the target VKr. In synchronous mode (default), the Subscribed Content Library automatically receives new VKrs from the Broadcom repo. In asynchronous mode, the admin validates and imports manually. Always verify the target OVA is locally present before any upgrade.
Step 2 — Edit the version in the Cluster manifest. Change topology.version from v1.30.0+vmware.1 to v1.31.0+vmware.1, then kubectl apply. CAPI triggers the upgrade sequence.
Step 3 — Control plane upgrade. KubeadmControlPlane upgrades CP nodes one by one with etcd validation at each step. If etcd doesn’t regain quorum after a node restart, the upgrade halts. That’s the right behavior — don’t force past it.
Step 4 — Worker upgrade. Once the control plane is stable, MachineDeployments start their rolling update. Pods are drained according to their PDBs. Stricter PDBs mean slower upgrades — that’s the point.
Step 5 — Post-upgrade validation. Smoke test of critical workloads, CRD verification (some are versioned), metrics check (the kubelet changes version, some metric labels too).
Skipping versions ∉ supported
You can’t skip a Kubernetes minor. v1.30 → v1.32 directly = guaranteed failure. You have to go through v1.31. Same for VKrs: the sequence respects the upstream Kubernetes compatibility matrix. Read the VKr release notes before each upgrade to validate the sequence.
Backup and restore
A topic often deferred. Don’t defer it.
What — snapshot of the etcd store containing all Kubernetes state (resources, secrets, statuses).
When — automatic with KubeadmControlPlane on configurable intervals. Stored on a dedicated PVC, to be exported regularly to external storage.
Restore — manual procedure, complex. Practice in non-prod before you need it in prod. Covers cluster state only, not workloads.
What — application-level backup at the Kubernetes layer (manifests, PVCs, CSI snapshots). The de-facto standard for Kubernetes backup.
When — deploy from production day 1. Regular schedule on namespaces with stateful workloads. Storage to S3 (on-premise object storage or cloud).
Restore — granular (per namespace, per resource), with or without PVCs. The primitive to know for migrating workloads between clusters or recovering from accidental deletion.
What — node VM snapshots at hypervisor level, via vSphere or third-party tools (Veeam, Rubrik, etc.).
When — useful for Supervisor VMs and control plane nodes. Avoid for workers — restoring a worker doesn’t make sense, CAPI replaces them more cleanly.
Restore — VM-level, not resource-level. Insufficient on its own for a Kubernetes cluster — combine with etcd snapshots and Velero.
Stated recommendation. Velero for application workloads, automatic etcd snapshots for cluster state, vSphere snapshots for Supervisor VMs. The three are complementary; none replaces the others.
Observability: the standard stack
The CNCF ecosystem is usable as-is on VKS, with integrations to VCF Operations to correlate infrastructure and workloads.
┌─ Workloads (pods) ─────────────────────────┐
│ ↓ metrics │
│ ↓ logs │
│ ↓ traces (optional) │
├────────────────────────────────────────────┤
│ Prometheus (metrics scrape) │
│ Loki (logs ingestion) │
│ OpenTelemetry Collector (traces) │
├────────────────────────────────────────────┤
│ Grafana (dashboards) │
│ Alertmanager (alerting) │
├────────────────────────────────────────────┤
│ forward → VCF Operations for Logs │
│ forward → VCF Operations for Apps │
└────────────────────────────────────────────┘Metrics. Prometheus Operator deployed via Helm. ServiceMonitor for workloads, PodMonitor for pods without Service. Default retention (15 days) is too short for post-mortem analysis past the sprint — bump to 30-90 days, and offload cold data to Thanos or Mimir.
Logs. Loki with Promtail as node-level collector. Simple architecture to operate, but watch chunk sizing (defaults are too conservative for clusters with many ephemeral pods). Forward critical logs to VCF Operations for Logs to correlate with vSphere events.
Grafana dashboards. The default Kubernetes dashboards (kubernetes-mixin) cover the essentials. Customize for application SLIs, not for infra metrics — those are better served by VCF Operations dashboards.
Alerting. Alertmanager for pod-level alerts. For infra alerts (host failure, full datastore), VCF Operations is better positioned. Avoid duplicating alert sources — golden rule: one alert, one source.
VCF Operations for Apps: the missing layer
VCF Operations for Apps (formerly Aria Operations for Applications, and Wavefront before that) is the component bridging Kubernetes metrics and vSphere metrics. For organizations already invested in the VCF Operations stack, this is the layer that justifies the ecosystem — it avoids maintaining two parallel observability systems.
Capacity management
The cluster grows. The cluster must also be constrained.
vSphere namespace quotas. CPU, memory, storage, number of clusters per namespace. Calibrate by team profile: a data team namespace ≠ a web team namespace. Quotas are the most effective governance tool against cost drift — and they’re free.
ResourceQuota and LimitRange inside the cluster. Within a cluster, ResourceQuota and LimitRange control consumption per Kubernetes namespace. Mandatory on multi-tenant clusters.
Capacity planning. Based on Prometheus metrics, model expected growth (pod count, namespace count, RPS) and anticipate adding capacity. Once a quarter is enough for most platforms — except during application growth phases when it should be monthly.
Gotchas and common traps
PDBs too strict = upgrades that never finish
Desynced Content Library = silently failing upgrades
etcd full disk = control plane down
Loki cardinality explosion
Prometheus storage saturating during upgrade
Velero PVC restore: namespaces must pre-exist
Upgrades on stateful workload clusters
Conclusion and what’s next
Key takeaways. Day-2 ops on VKS are framed by CAPI: everything goes through the manifest, never through direct edits. Kubernetes upgrades are decoupled from the Supervisor but respect the upstream matrix — no skipping minors. Velero is non-negotiable for application workloads. The standard observability stack works well and integrates with VCF Operations. Gotchas are rarely in individual components, more often in their interactions (PDBs + upgrades, cardinality + Loki, etcd + Custom Resources).
What’s next. The final article in this series covers GitOps on VKS: how to automate application deployment with Argo CD, manage secrets with External Secrets Operator, and extend the pattern to multi-cluster. That’s the step that turns an operated cluster into a self-service platform.
Resources.
Going further:
- Broadcom TechDocs — Managing VKS Clusters — operational reference
- Cluster API Book — upstream CAPI docs, mandatory reading
- Velero Documentation — Kubernetes backup/restore
- Prometheus Operator — deployment and configuration
- Loki Documentation — operational guides
- VCF Operations — platform observability integration
Get the next one by email
New articles and series, sent when they are published. No other mail.
