Table of contents
At the coffee corner, a colleague shows TOP the order that fails from her phone. The internal menu correctly shows zero coffee beans, but the reorder button still does not work. TOP records this request’s context before following it through DNS and the Service, then checking whether the API can reach PostgreSQL.
Before deciding the app is wrong, he needs to follow the request: the right name, the right destination, a ready backend, and a response he knows how to interpret.
In episode one, we distinguished the management API from the workload cluster. Now we follow GET /menu to that cluster’s application. TOP Dev uses the application’s behavior to guide the investigation, then connects it to infrastructure evidence.
Revised on 27 October 2026, this article separates common Kubernetes concepts from the actual VKS network design. An NSX/VPC-backed environment and a VDS-backed one need their own supported topology. The exercise commands target a disposable workload namespace described in the next episode; they are not instructions to modify a production network.
Write the starting point on the test result
“Coffee Break works from my laptop” is incomplete evidence. Which address, port, protocol and path did the laptop use? Did it traverse the external gateway, connect through port-forward or reach a Pod directly? Was TLS verified? Each answer changes what the success proves.
TOP records the source environment, destination name, resolved address, port, release and response. He uses two paths: a request inside the application cluster and a request from the intended user network. They answer different questions.
| Observation | Useful next check |
|---|---|
| Name does not resolve | Resolver used, search domain, DNS record and permitted DNS traffic |
| Name resolves, TCP times out | Actual destination, route, filtering, return path and service health |
| TLS verification fails | Certificate chain, name, trust and the real termination point |
| HTTP 404 | Host/path routing and the handler receiving the request |
| HTTP 503 | Which component returned it and which upstream dependency is unavailable |
| HTTP 200 with the wrong list | Actual release, request target, configuration and data source |
A timeout does not identify a firewall by itself. An HTTP status does not identify its issuer by itself. Preserve the response and relevant logs before changing rules. The upstream DNS debugging guide helps establish which resolver and name are involved.
A Service needs matching, usable destinations
Inside our workload cluster, the coffee Service selects Pods labeled app: coffee. Kubernetes records backend endpoints through EndpointSlices. Readiness influences whether a backend is normally eligible for service traffic. This links three observations: the Service’s selector and port, the Pod’s labels and readiness, and the published endpoints. Debug Services.
Logical selection and endpoint discovery. The drawing does not prescribe a proxy, CNI implementation or physical route.
With the disposable example installed and the kubeconfig context named coffee-workload:
kubectl --context coffee-workload -n coffee-demo get service coffee -o yaml
kubectl --context coffee-workload -n coffee-demo get pods -l app=coffee --show-labels
kubectl --context coffee-workload -n coffee-demo get endpointslices -l kubernetes.io/service-name=coffee -o yaml
Check port, targetPort, labels, endpoint addresses and readiness together. A Pod in phase Running can still be unready. A selected process may listen on a different port. A Service with no suitable destinations cannot manufacture an application response.
In the small exercise, this call runs in one of the existing coffee Pods and uses the internal Service name:
kubectl --context coffee-workload -n coffee-demo exec deployment/coffee -- wget -qO- http://coffee/menu
Expected output lists coffee-beans with stock 0 and filters with stock 24. The image includes wget; another application image may not. Exec must be allowed by your role. A diagnostic Pod with different labels or identity can be treated differently by policy, so describe its context rather than assuming equivalence.
At the diagnostic board, TOP and his colleague connect their observations to the request path: DNS resolution, Service selector and EndpointSlices. One line is crossed off in the notebook: that check passed. They still need to identify the failing hop.
External access adds a separate arrangement
A ClusterIP Service is an internal abstraction. For a user outside the cluster, the platform supplies an approved exposure mechanism. A LoadBalancer Service delegates provisioning to the installed integration. HTTP host/path routing requires an implemented gateway or ingress arrangement. Merely creating an API object does not install the controller that fulfills it. Kubernetes Service types.
The former article drew Avi/AKO, T0/T1 and direct Pod routing as one mandatory sequence. That is too broad. Identify the actual load-balancer provider and backend mode: it may forward through node-level targets, another supported integration or the platform’s configured routing. Do not infer direct routability of every Pod from the presence of NSX.
For NSX/VPC, document the published subnet, gateway, allocation, exposure and egress policies. For VDS, document the workload network and supported load-balancing arrangement. On either path, the application contract still needs addresses, ports, ownership and a return route. Broadcom groups these choices and policies in its VKS networking guide.
The boxes marked “configured” are questions to resolve against your environment. They are not a claim that every VKS deployment uses the same load balancer or NAT.
For the external test, use the real service FQDN from the intended client network. Verify HTTPS normally, including its trusted CA. Record where TLS terminates and whether the next hop is encrypted. A valid certificate on the gateway does not establish the database’s trust configuration.
The app’s outgoing request starts another journey
A persistent version of Coffee Break could, for example, call PostgreSQL on TCP 5432. That connection originates from its execution environment, not the user’s browser. The observed source may be translated. Ask the network team which address the database firewall actually sees and how the reply returns.
Node addresses, Pod CIDRs, Service CIDRs and external address pools serve different purposes. Reserve and validate them through the network design; check overlap with reachable corporate networks and the supported platform rules. Do not turn one example CIDR into a universal allocation plan.
A standard Kubernetes NetworkPolicy can select ingress and egress permissions when a compatible network implementation enforces it. Policies can be additive, and isolated ingress/egress must both permit the relevant path. They cannot override a denial in another firewall. See Network Policies.
TOP starts with the application needs: DNS, approved destinations and ports. The platform translates them into the right controls. Applying a default-deny policy without planning DNS and dependencies can break a perfectly healthy process. Conversely, an allow-all policy hides which flows the service actually needs.
Keep a before/after network acceptance sheet
Before exposure or a network change, record the Service selector and ports, ready endpoints, image release, intended DNS/TLS behavior and one successful menu response. Also capture the application-to-database test if that dependency exists. The static fixture in our exercise deliberately has no database; its success cannot prove that separate dependency.
After the change, repeat the same requests from the same points. Compare the response contents as well as the status. If internal access works and external access fails, investigate the additional external arrangement with its owner. If internal endpoints disappear, start with labels and readiness before asking for a BGP change.
Port-forward is helpful to inspect a selected application process, but bypasses several parts of the normal service path. Treat its success as scoped evidence. In the next episode, we will intentionally change the demo Service selector, observe the missing backends, then restore it. The Pod count will stay reassuringly unchanged throughout the fault.
Beside the storage cupboard, his colleague shows him the empty jar: the menu’s zero stock makes sense, but they still need to place an order. TOP keeps one precise network question in his notebook: which hop between the browser, API Service and PostgreSQL is failing?
Continue with the first cluster and first application. We will keep the two contexts explicit and make a small service work before adding a public address or a database.
Get the next one by email
New articles and series, sent when they are published. No other mail.



