Table of contents
At the team calendar, TOP marks the maintenance window while his colleague reviews the checklist. “Before the window, let’s record the menu version and one confirmed order identifier for comparison.” Coffee Break works today; we need to know what we’ll observe during the operation and afterward.
The application contract gives us two concrete checks to preserve: the static menu version and a confirmed order identifier. A completed maintenance task and a useful service require different evidence.
We now have a first application and a deliberate fault we can explain. Day two begins when we must change its environment while respecting the service contract. TOP Dev follows the effect on requests and data, while the platform team operates the supported lifecycle workflow.
This article was revised on 27 October 2026. The sequence is a preparation and acceptance framework, not a claim of a performed VKS upgrade. The small stateless fixture from episode three helps explain the Service path; production applications, especially databases, need additional recovery and consistency tests.
Three version decisions, then one supported route
Separate the platform/Supervisor version, the installed VKS service version and the workload cluster’s Kubernetes release. Record the currently installed values and the intended targets before planning order. Independent lifecycles still have compatibility boundaries.
Check the official interoperability matrix and release notes, including existing workload clusters and installed add-ons. A change to the VKS service is a different operation from changing a workload cluster release. A supported route may require intermediate steps; do not turn a guessed version string into an upgrade plan. Broadcom KB 455622.
For environments upgrading vCenter to 9.1.0 or 9.1.1, include the targeted pre-upgrade advisory published by Natalie Fisher on 9 October 2026. It identifies a risk to Supervisor control-plane VMs and Foundation Load Balancers when an earlier vLCM/VUM depot reset left database inconsistencies. Review its source-version scope and required validation/remediation with the platform owner. The repair scripts belong to the official procedure; they are not an application-team exercise. Advisory and linked KBs.
That advisory does not mean every application deployment or workload-only patch has the same trigger. It means a blanket “everything is independent” statement would omit a material platform precondition.
Reconcile the source of truth, not the resulting VM
A VKS cluster specification is reconciled into node infrastructure through managed controllers. If the workflow uses managed topology, edit its supported source or approved template rather than independently changing generated MachineDeployments or VM hardware. A portal or Git repository may own that source; identify which one before editing anything.
Identify the change before selecting the source to edit:
| Change | Source to edit | Effect to verify |
|---|---|---|
| Application replicas | Application Deployment | Additional Pods and eligible placement on existing nodes |
| Worker capacity | Cluster specification or approved template | Node addition/removal and possible Pod movement |
| VM class | Source supported by the platform workflow | Possible worker replacement |
Ask what the proposed field triggers in the installed release; the result depends on the workflow and topology.
Class rebasing can also have consequences. KB 439754 explains a case where retaining an older ClusterClass is valid and moving to a new schema can trigger a rollout. A newer-looking class name is not a maintenance objective by itself.
Spare capacity and disruption rules work together
A node replacement can need temporary CPU, memory, IP addresses and storage beyond the steady-state target. The application also needs somewhere eligible to reschedule. Resource requests, node selectors, affinity, topology spread, taints and volume constraints can prevent a new Pod from using apparently free capacity.
The demo Deployment has two replicas, but does not promise that they are on separate nodes or failure domains. A production design must express the intended placement and verify it. Two processes on the same failed node do not provide the desired separation.
A PodDisruptionBudget, or PDB, constrains covered voluntary evictions. For a suitable replicated service, an example budget is:
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
name: coffee
namespace: coffee-demo
spec:
minAvailable: 1
selector:
matchLabels:
app: coffee
This is an optional example to review against the service. It does not create replicas, reserve a worker or prevent an involuntary failure. It also does not govern a Deployment’s rolling-update behavior in the same way as eviction-based node drain. Its selector must cover the intended Pods. Disruptions and PDB configuration.
Before maintenance, inspect budgets and scheduling constraints. A blocked eviction is useful information: the requested operation cannot currently honor that budget. Investigate unavailable replicas, placement, capacity and application recovery. Blindly removing the budget may exchange a stuck upgrade for a service outage.
The stop decision exists during the change. The last green infrastructure task is not the only acceptance gate.
Preserve the data we would need after a rebuild
The menu fixture is stateless configuration. Reapplying its manifest recreates the same GET response, not an order. The reference app stores stock and orders in shared PostgreSQL; recreating its Deployment or cluster does not recreate the last valid business state.
TOP separates the recovery layers:
| Layer | What must be recoverable | Proof expected |
|---|---|---|
| Platform management | Supported configuration and management-component recovery | Platform-owned procedure matched to the version |
| Workload definition | Reviewed manifests, required APIs/add-ons and configuration | Rebuild onto a compatible target |
| Persistent data | Database/PV contents with an agreed consistency point | Restore and useful read/write validation |
| Identity and dependencies | Secrets, CA trust, permissions, DNS and external services | Restored app can perform the intended action |
TOP opens the restored menu configuration and checks its contents. For the reference app, he also asks the data owner to verify a restored order record in PostgreSQL. “A backup job is one signal. A readable restored record is evidence.”
For VKS application backup, choose the supported method and compatible components described in the Broadcom Velero guide. Confirm what it captures, where copies reside, who can read them, and how volume data is protected. A successful object backup alone does not prove usable database contents.
Velero’s disaster-recovery documentation describes restoring into a prepared target. Practice that target preparation and the application’s own validation. Where snapshots are used, understand their consistency and storage dependencies; do not equate an arbitrary VM snapshot or a retained snapshot on the same system with an independent, tested backup.
Observe the change from infrastructure to request
Before the window, assign a change owner, a decision deadline and the recovery route for the reached state. Capture baseline request success, latency, errors, actual image and relevant data checks. Decide which failures stop progression and reserve enough time to act on them.
During the supported lifecycle operation, correlate its task or reconciliation condition with node readiness, Pod placement, restarts, volume attachments and user-visible symptoms. Do not delete a Machine or revert a node VM simply because a rollout is slow. That may interfere with reconciliation or data consistency. A release-specific incident needs its applicable support procedure.
For workload resources with the intended read permissions:
kubectl --context coffee-workload -n coffee-demo get deploy,pods,pdb -o wide
kubectl --context coffee-workload -n coffee-demo get events --sort-by=.metadata.creationTimestamp
kubectl --context coffee-workload -n coffee-demo get endpointslices -l kubernetes.io/service-name=coffee
Nodes and detasseed management conditions may require platform-team access. Bring the timestamp and application symptoms to that team rather than granting every application owner broad cluster access.
Use logs, metrics and, where available, traces to connect failures with operations. A useful menu signal includes request errors and duration, not just host CPU. Choose collectors and backends supported by the installed environment; avoid assuming a historical chart or collector remains the default. The Prometheus instrumentation guide gives the application perspective on measuring useful behavior.
After the operation, repeat the baseline from the same relevant network and identity. Verify actual versions, reconciliation health, expected replicas and endpoints, persistence and external dependencies. Test a small authorized write/read cycle when the service contract requires writes. Check new backups and alerts, then retain the evidence and any remaining deviation.
The before-and-after static menu responses are kept with their release label and quantities. TOP also has the confirmed order identifier checked after maintenance; the fixture cannot prove order durability. At handover, he passes these observations to his colleague so she can understand what was checked and what still needs watching.
The result we hand to the next release
A completed task says the infrastructure operation ended. Acceptance says the agreed service behavior has been demonstrated, including recovery readiness. They are related records with different evidence.
For the fixture, our evidence stays deliberately small: the selected image, two ready replicas, usable Service endpoints and the static menu response with its release and quantities. For production, add placement, load, data consistency, authentication and restoration results. If an infrastructure downgrade is unsupported, a Git revert cannot turn it into a supported rollback; the recovery plan must reflect that boundary.
The final episode follows GitOps with Argo CD: how to make the reviewed application plan drive reconciliation without confusing a green synchronization status with a successful menu request.
Get the next one by email
New articles and series, sent when they are published. No other mail.



