Being able to say, for any packet, what touches it and in what order: pod, CNI, Service, kube-proxy or its eBPF replacement, DNS, policy, ingress. Most delivery and incident tasks in the other four domains eventually collapse into this one.
make upmake gitea gitopsmake sec
Orientation
Networking is the substrate every other domain stands on. A Crossplane XR that never goes Ready, an Argo CD app stuck Progressing, a Prometheus target that will not come up, a canary that never receives traffic: a surprising share of those are one Service selector, one missing DNS egress rule, or one unready endpoint.
Not "explain CNI". It asks you to make traffic work or make traffic stop, then prove it: expose a workload, restrict a namespace, author a Gateway and an HTTPRoute, or explain why a Service resolves and then refuses connections. Every one of those is a five-minute task if the model in your head is exact, and a twenty-minute flail if it is fuzzy.
The four rules of the Kubernetes network model
- Every pod gets its own IP, cluster-routable, no NAT between pods.
- Pods on a node can reach all pods on all nodes without NAT.
- Agents on a node (kubelet, system daemons) can reach all pods on that node.
- A pod sees its own IP as the same address other pods use to reach it.
The CNI plugin is whatever implements those rules: routing, overlay, or eBPF datapath. Everything above (Services, DNS, policy, Gateways) is built on the assumption that the four rules already hold. When they do not, nothing above them behaves sanely, which is why "is it a CNI problem or a Service problem" is the first fork in any network diagnosis.
Services and the thing that actually makes them work
ClusterIP is a virtual IP that exists only as translation rules on each node; nothing listens on it, no interface owns it, you cannot ping it in any meaningful sense. NodePort opens the same Service on a high port (30000–32767 by default) of every node. LoadBalancer is NodePort plus something external handing out a real IP (in this lab, cloud-provider-kind). Headless (clusterIP: None) skips the VIP entirely and returns pod IPs straight from DNS, which is what StatefulSets need for stable peer discovery. ExternalName is a CNAME with no proxying at all, and it exists mostly to trip you up in a multiple-choice question you will never see.
| Type | spec bits | Reaches it from | Use it when |
|---|---|---|---|
| ClusterIP | default | inside the cluster | the 90% case; every internal call |
| NodePort | nodePort: 3xxxx | any node IP | bootstrapping, bare metal, demos |
| LoadBalancer | type: LoadBalancer | outside | real external entry point (loadBalancerClass only picks between implementations) |
| Headless | clusterIP: None | DNS returns pod IPs | StatefulSet peers, client-side LB |
| ExternalName | externalName: host | DNS CNAME only | aliasing an out-of-cluster host |
The thing that makes a Service work is not the Service. It is the EndpointSlice behind it, and the selector that fills it. A Service with no ready endpoints resolves fine and then refuses connections, which is one of the most common exam-shaped failures. Check order, every time: selector matches pod labels → pods are Ready (readiness gates endpoint membership) → kubectl get endpointslices -l kubernetes.io/service-name=<svc>.
Two related fields decide whether traffic reaches a pod at all. readinessProbe failure pulls the pod out of the slice, which is the intended way to drain a pod. publishNotReadyAddresses: true overrides that and is how headless Services for clustered databases let peers find each other before they are serving. And terminationGracePeriodSeconds plus a preStop sleep is the standard trick for the race where a pod is deleted but nodes have not yet removed its rules; connections refused during rollouts almost always trace here.
Traffic policies, which cost people points
externalTrafficPolicy: Cluster(default) SNATs and may hop to another node, so the backend sees the node's IP, not the client's.Localpreserves the client source IP and only routes to pods on the receiving node, with the trade that a node holding no pod blackholes the traffic (health checks are what stop the external LB from sending there).internalTrafficPolicy: Localis the same idea for in-cluster traffic: node-local endpoints only. Used for node-local caches and log shippers.sessionAffinity: ClientIPis the only affinity a Service offers. Anything richer belongs in a mesh or gateway.
Five conditions, one request. Turn any of them off and see which hop drops it, what the user reports, and the command that proves it:
kubectl get endpointslices -l kubernetes.io/service-name=X beats describe svc, because it shows you conditions per address (ready, serving, terminating) rather than a summarised list. When a rollout half-breaks, those three booleans tell the whole story.
The datapath: kube-proxy, eBPF, and this cluster's choice
Classic kube-proxy watches Services and EndpointSlices and programs the node: iptables mode writes DNAT chains (simple, and linear-ish to rule count), IPVS mode uses kernel load balancing with real scheduling algorithms. eBPF datapaths (Cilium, Calico) replace kube-proxy entirely, doing the translation in the socket or TC layer, which removes the rule-table scaling problem and unlocks flow visibility.
client pod │ connect 10.96.0.42:80 ← ClusterIP, exists only as a rule ▼ [ datapath ] kube-proxy iptables/IPVS or eBPF (Cilium, this lab) │ DNAT → picks one endpoint from the EndpointSlice ▼ backend pod IP 10.244.2.17:8080 │ ▼ policy is evaluated on thepod identity, not the Service VIP
Note the last line. This is why an ipBlock rule naming a ClusterIP never matches: by the time policy is evaluated, the destination has already been rewritten to a pod IP. That single fact resolves a whole family of "my NetworkPolicy does nothing" tasks, and it is exactly the trap section 3.3 springs on you with CloudNativePG.
This cluster runs Cilium instead of kindnet plus kube-proxy for one load-bearing reason: kindnet does not enforce NetworkPolicy at all. A lab where policies apply cleanly and change nothing teaches you the wrong lesson permanently. Cilium also brings identity-based policy (labels are compiled into numeric identities, so policy survives pod IP churn) and Hubble, which shows flows with their verdict: forwarded or dropped, and by which policy.
CiliumNetworkPolicy adds what upstream NetworkPolicy cannot express: toFQDNs (allow api.github.com by name, enforced at DNS), toEntities (kube-apiserver, world, host, remote-node), and L7 rules for HTTP/DNS/Kafka. The exam will not test CRD field names, but "which policy engine can express egress to the API server" is a fair scenario, and the answer here is toEntities: [kube-apiserver].
DNS: the layer that fails quietly
Names are <svc>.<ns>.svc.cluster.local, served by CoreDNS in kube-system, backed by a Service called kube-dns (the name is a fossil; the pods are CoreDNS). Pods get a /etc/resolv.conf with a search list and ndots:5, which means any name with fewer than five dots is tried against every search domain first. backend becomes four or five queries before the right one lands; a trailing dot (backend.default.svc.cluster.local.) skips the search list entirely, and that is the cheap fix for DNS-heavy workloads.
| Record | Resolves to | Notes |
|---|---|---|
| svc.ns.svc.cluster.local | ClusterIP | the normal A record |
| svc.ns | ClusterIP | works via search domains |
| headless.ns.svc… | all ready pod IPs | multiple A records, client picks |
| pod-0.headless.ns.svc… | one pod | StatefulSet stable identity |
| _port._tcp.svc.ns.svc… | SRV record | named ports; how peers discover ports |
spec.dnsPolicy and dnsConfig on a pod let you override all of this: ClusterFirst (default), None plus explicit nameservers, Default (inherit the node's). It comes up when a workload must resolve an external private zone.
team-a allows DNS egress through one explicit NetworkPolicy rule (UDP/TCP 53 to kube-dns). The netpol break drill removes exactly that rule. Result: every pod stays Running, every probe stays green, and nothing can resolve anything. No pod listing will ever show it. You find it by making something try (nslookup from inside) and then reading Hubble's DROPPED verdicts on port 53. Filed away now, it costs you 90 seconds in section 4.6 instead of seven minutes.
NetworkPolicy semantics, exactly
- Policies are namespaced and select pods, never Services.
- A pod is "isolated" for a direction the moment any policy selects it for that direction. Until then, everything is allowed.
- Rules are additive allow-lists. There is no deny rule. You loosen by adding, and you tighten only by removing, an asymmetry the break drill exploits, because a missing rule looks exactly like a rule that was never there.
- Ingress and egress are independent. A cross-namespace call needs an egress allow on the caller's side and an ingress allow on the callee's. Discovering that empirically once (exercise 3 below) saves you re-deriving it under pressure.
- Default-deny is itself a policy: empty
podSelector,policyTypes: [Ingress, Egress], no rules. - Selector scoping inside a rule matters:
namespaceSelectorandpodSelectorin the same list item is an AND (those pods in those namespaces); as two separate items it is an OR. This is the single most common authoring bug.
A task that says "team-a must reach only team-b's web service and DNS" is asking for three objects, not one: default-deny in team-a, an egress allow in team-a, an ingress allow in team-b. Grade yourself the way a grader would: a curl that returns 200 and a second curl to something else that times out.
Ingress vs Gateway API
Ingress was one object owned by nobody in particular, extended by a swamp of controller-specific annotations. Gateway API replaces it with a role-split model, and the roles are the point:
| Kind | Owned by | Says |
|---|---|---|
| GatewayClass | infrastructure provider | which controller implements Gateways of this class |
| Gateway | platform team | listeners: ports, protocols, TLS, and which routes may attach |
| HTTPRoute / GRPCRoute / TCPRoute | application team | match rules, filters, weighted backends |
| ReferenceGrant | the namespace being referenced | consent for a cross-namespace backend reference |
Route attachment is two-sided, in the same spirit as NetworkPolicy: the route names a parentRef, and the Gateway's listener declares allowedRoutes (same namespace, selected namespaces, or all). Both must agree, and a route whose attachment is refused reports it in status rather than failing to apply.
Status conditions are how you grade your own work: Accepted (the controller understood it), Programmed (the data plane is configured), ResolvedRefs (backends and secrets exist and are permitted). Reading those three beats guessing every time.
This cluster installs the standard-channel Gateway API CRDs so you can author and schema-validate all of it, but no controller programs a data plane for them on the main cluster. Your Gateway will sit unprogrammed, and I would rather say so than let you believe a Gateway "worked". On the mesh cluster (make mesh), Istio serves Gateway API for real: the same manifests, an actual Programmed condition.
One neighbour the Gateway needs and Kubernetes does not provide: certificates. A TLS listener references a Secret, and something has to keep that Secret valid. In practice that something is cert-manager: an Issuer/ClusterIssuer (ACME, CA, or Vault) plus a Certificate, or the cert-manager.io/cluster-issuer annotation on the Gateway, which issues into the Secret and renews before expiry. Not installed in this lab, but "who renews that certificate" is a fair question about any Gateway you design, and ResolvedRefs: False on a listener is usually its absence.
Progressive delivery ties in here (section 2.5): weighted backendRefs on an HTTPRoute are how a mesh shifts traffic by route weight instead of by replica count, and Flagger drives exactly those weights.
Exercises
With gitops up, Argo CD sits behind a LoadBalancer (running server.insecure, so plain http):
kubectl -n argocd get svc argocd-server -o wide # note the EXTERNAL-IP
kubectl -n argocd get endpointslices -l kubernetes.io/service-name=argocd-server
curl -s -o /dev/null -w '%{http_code}\n' http://<EXTERNAL-IP>The EXTERNAL-IP is real only while cloud-provider-kind runs; without it, kubectl -n argocd port-forward svc/argocd-server 8080:80 and curl localhost:8080 instead, and the rest of the exercise is unchanged. Now break it in a way you can explain:
kubectl -n argocd patch svc argocd-server --type=merge -p '{"spec":{"selector":{"app":"nope"}}}'Watch the endpointslice lose its endpoints and curl start failing while DNS still resolves. Then look at what your patch actually did before undoing it: kubectl -n argocd get svc argocd-server -o jsonpath='{.spec.selector}' shows the original two labels plus app: nope, because a JSON merge patch merges maps rather than replacing them. So the honest restore is removing your key, not re-adding theirs: --type=merge -p '{"spec":{"selector":{"app":null}}}'.
app.kubernetes.io/name: argocd-server and app.kubernetes.io/instance: argocd, and curl returns 200. Two lessons for the price of one; the merge-patch semantics come up on their own exam tasks.Run a probe that policy must kill:
cilium hubble port-forward &
kubectl -n team-a run probe --image=curlimages/curl:8.11.1 --restart=Never \
-- curl -s -m 5 http://example.com
hubble observe --namespace team-a --verdict DROPPED --last 20Write a Gateway named web using listener port 80, protocol HTTP, plus an HTTPRoute that matches path /demo and backends a Service demo:80. Don't copy from docs; build it from kubectl explain gateway.spec.listeners and kubectl explain httproute.spec.rules.
kubectl get gateway web -o yaml shows the spec you meant. Status stays unprogrammed here; on the mesh cluster Istio would accept the same manifests.From a pod in default, resolve short and long names:
kubectl run dnsprobe --image=busybox:1.37 --restart=Never -it --rm -- \
sh -c 'nslookup argocd-server.argocd && nslookup argocd-server.argocd.svc.cluster.local'Then repeat inside team-a and explain why it still works (the tenant policy allows port 53 to kube-dns explicitly; find that rule in examples/multitenancy/team-a.yaml).
Self-check
A Service resolves in DNS but every connection is refused. Name the three things you check, in order.
Selector against actual pod labels; pod readiness (an unready pod is not in the slice); the EndpointSlice itself. If the slice has addresses with ready: false, it is a probe problem, not a Service problem. If the slice is empty, it is a selector or a scheduling problem.
Why does a NetworkPolicy ipBlock naming a Service's ClusterIP never work?
Because DNAT happens before policy evaluation: by the time the packet is checked, its destination is a pod IP. Policy is expressed in pod selectors and identities, not virtual IPs. For the API server specifically, Cilium's toEntities: [kube-apiserver] is the expressible form.
Your backend needs the real client IP. What do you change, and what breaks?
externalTrafficPolicy: Local on the Service. The trade: nodes with no backing pod stop serving that traffic, so you now depend on the load balancer's health checks to avoid them, and you lose the even spread that Cluster gave you.
What is the difference between a headless Service and a ClusterIP Service with one endpoint?
Headless has no VIP and no datapath translation: DNS returns pod IPs directly and the client chooses. A one-endpoint ClusterIP still goes through the VIP and its rules, so the client never learns the pod IP. StatefulSets need the former for stable per-pod names (pod-0.svc…).
Team-a can curl team-b's Service after you added an egress allow, but it still fails. What did you forget?
Team-b's ingress side. Both ends must open; policies are additive allow-lists evaluated independently at each pod. Also check DNS: if egress is now restricted in team-a, resolving web.team-b.svc needs its own port-53 allow.
An HTTPRoute applies cleanly but nothing routes. Where do you look first?
status.parents[].conditions on the route: Accepted false usually means attachment was refused by the Gateway's allowedRoutes; ResolvedRefs false means a backend Service or a TLS secret is missing (or needs a ReferenceGrant across namespaces). On the Gateway, Programmed false means no data plane, which is the permanent state on this lab's main cluster.
Docs to know your way around
- kubernetes.io: Services, DNS for Services and Pods, Network Policies. The DNS page's search-domain section is the one people never read.
- gateway-api.sigs.k8s.io: the API model page with the role diagram; the "route attachment" section.
- docs.cilium.io: Hubble observe reference; CiliumNetworkPolicy entities and FQDN rules.
- Offline, in the exam:
kubectl explain service.spec,kubectl explain networkpolicy.spec.egress,kubectl explain httproute.spec.rules --recursive. Faster than docs and always version-correct for the cluster in front of you.