examples/multitenancy/team-a.yaml is the whole syllabus for this section in 75 lines: one namespace carrying every guardrail the exam can ask about. Read it top to bottom before doing anything else; every exercise below pokes at one of its objects.

needsmake up sec

Orientation

competency 1.3 · optimizing multi-tenancy resource usage

A namespace is a folder. A tenant is a namespace with a stack of controls attached. Each one closes a different hole, and each one can be the thing that just refused your pod. The competency's word is "optimizing", which is the giveaway: the exam cares about isolation that does not waste capacity, and about knowing which control refused you.

The soft/hard spectrum

Multi-tenancy is not a switch. Soft: shared control plane and nodes, separation by namespace, quota, policy and RBAC: this lab, and most platforms. Harder: dedicated node pools via taints (no shared kernel), then virtual clusters (own API server, e.g. vcluster), then separate clusters. Each step up costs idle capacity and operational surface. Being able to name the ladder and say what each rung buys is a complete answer to any "how would you isolate these tenants" question.

The isolation stack, control by control

what refuses you, and with which message
  1. ResourceQuota caps the namespace total: requests.*, limits.*, object counts (pods: "20", services.loadbalancers: "1"), and storage per class. Crucially, a quota that lists requests.cpu or limits.memory rejects any pod that fails to declare them, which leads directly to point 2.
  2. LimitRange fills in per-container defaults (default, defaultRequest) and bounds (min, max, maxLimitRequestRatio), so a bare pod arrives at the quota check with numbers already attached. LimitRange defaulting is what keeps tenant onboarding friction-free while the quota stays enforceable.
  3. NetworkPolicy default-deny plus explicit allows. team-a allows same-namespace traffic and DNS to kube-dns, nothing else. Additive allow-lists mean you loosen by adding policies and tighten only by removing rules, an asymmetry the netpol break drill exploits.
  4. Pod Security Standards labels on the namespace (enforce baseline, warn and audit restricted here). Covered properly in section 5.3.
  5. RBAC scoping: a Role and RoleBinding giving user dev-a real rights inside team-a and none outside. Covered properly in section 5.1.
Admission order decides who refuses you

The order is: mutating admission (LimitRanger injects defaults, Kyverno mutates labels) → validating admission (PSS, quota, Kyverno/Gatekeeper validate). So a pod with no resources block can be admitted because LimitRange gave it some, and a validating policy demanding requests will never see a violation in a namespace that has a LimitRange. Section 5.2 makes a whole lesson of this; here, just internalize that a mutation can satisfy a validation the user never wrote.

Reading the refusal

Message containsWho refusedFix direction
exceeded quota: team-a-quota, requested: …, used: …, limited: …ResourceQuotasmaller requests, fewer replicas, or a bigger quota
maximum cpu usage per Container is 500m, but limit is 2LimitRangefit inside the bounds
violates PodSecurity "baseline:latest"PSS admissionfix the securityContext
admission webhook "…kyverno…" denied the requestpolicy enginesatisfy the policy or exempt properly
is forbidden: User "dev-a" cannot create …RBACa rule, a binding, or the wrong namespace

Being able to name the refusing control from one line of error text is the difference between fixing the right object and guessing.

Do the arithmetic the way the admission controller does. Push the replicas up and see which line binds first, and what changes when the tenant declares resources instead of letting the LimitRange fill them in:

Quota's second-order effects

Quota is enforced by an admission controller against a usage cache, which is why the failure often appears one level down: a Deployment applies fine and its ReplicaSet logs the quota error while replicas stall below the target. Look at kubectl describe rs or namespace events, not at the Deployment. The same indirection shows up for PSS (5.3), and recognizing the two-level pattern generalizes across the whole exam.

Fairness beyond quota

the "optimizing" half of the competency

Quota bounds the worst case. It does not make sharing efficient, and four other mechanisms carry that load:

  • Priority and preemption. A PriorityClass decides who gets evicted when the cluster is full. Platform components (ingress, monitoring, CNI) should outrank tenant workloads; a tenant that can preempt your metrics stack is a tenant that can blind you. Quota can be scoped per priority class (scopeSelector) so nobody hoards the high-priority lane.
  • Overcommit ratios. Setting limits well above requests across every tenant is how you get density; it is also how you get a node that OOMs under coincident spikes. The lab's LimitRange (50m request / 200m limit) is a deliberate 4× burst ratio: a decision, not an accident.
  • Node pools with taints. When two tenants must not share a kernel (or a GPU, or a license), taint the pool and pair it with nodeAffinity. Cost: stranded capacity in each pool.
  • Idle reclamation. The unused space between requests and usage is the tenancy tax, and it is exactly what OpenCost puts a number on in section 1.5. "Optimizing multi-tenancy resource usage" is, in practice, driving that number down without breaking the guardrails above.

Two more things a real tenant needs that people forget in exam prep: a default ServiceAccount with nothing attached (so workloads do not inherit power), and namespaced quota on objects that cost money outside the cluster: LoadBalancers, PVCs, NodePorts. That is why team-a caps services.loadbalancers at 1: quota as a cost tool, not just a fairness tool.

Quotas beyond requests and limits

  • Object counts: count/<resource>.<group> works for any kind, including custom resources (count/widgets.example.com), and the specialized forms pods, services, secrets, configmaps, persistentvolumeclaims, services.loadbalancers, services.nodeports. These protect the control plane and the bill, not the nodes.
  • Scopes: scopeSelector narrows a quota to BestEffort/NotBestEffort, Terminating/NotTerminating (pods with activeDeadlineSeconds), CrossNamespacePodAffinity, or PriorityClass with In a list of class names. A quota scoped to PriorityClass In [platform-critical] with pods: "0" is how you stop tenants using the platform's own lane.
  • Per-class storage: <class>.storageclass.storage.k8s.io/requests.storage and .../persistentvolumeclaims, as in 1.3.
  • Ephemeral storage: requests.ephemeral-storage and limits.ephemeral-storage; container logs count against it, which is why a chatty pod can be evicted for disk.

The "optimizing" levers in field names: LimitRange.spec.limits[].maxLimitRequestRatio bounds overcommit per container (a ratio of 4 means a limit may be at most four times the request); PriorityClass plus preemptionPolicy decides who yields when the shared pool is full; a PriorityClass-scoped quota decides how much of the high lane each tenant may buy; and OpenCost's efficiency per namespace (1.5) tells you which tenant's requests are the hoarders. Bin-packing itself is the node autoscaler's job (1.2), and it only ever sees requests.

Tenancy models and the tools that implement them

namespaces-as-a-service · virtual control planes · clusters · HNC, Capsule, vcluster

The kubernetes.io multi-tenancy page splits isolation into control plane (namespaces, RBAC, quota) and data plane (network policy, storage, sandboxing, node isolation), and names two ways to share a cluster: a namespace per tenant or a virtual control plane per tenant. Scenario questions use exactly those words.

ModelTenant getsCannot isolateCostTools
Namespace per tenant (namespaces-as-a-service)one or more namespaces with quota, LimitRange, NetworkPolicy, PSS, RBACanything cluster-scoped: CRDs, StorageClasses, webhooks, cluster roles, API versionsnear zeroplain Kubernetes; HNC or Capsule to manage many
Virtual control plane per tenantits own API server, CRDs, cluster-scoped objects; pods still run on shared nodesthe kernel and node-level attacks; cross-tenant sharing gets harderone control plane pod per tenantvcluster
Cluster per tenanteverything, possibly its own hardwarenothing, but nothing is shared eitherfull duplication, fleet managementCluster API, managed clusters

The three tools by name

  • Hierarchical Namespace Controller (HNC): namespaces gain a parent; RBAC, NetworkPolicy and other objects propagate from parent to children (label hnc.x-k8s.io/inherited-from), and a tenant with rights in a parent can create subnamespaces (kubectl hns create dev -n team-a) without cluster permission to create namespaces. Every namespace gets tree labels (team-a.tree.hnc.x-k8s.io/depth) that NetworkPolicy selectors can use to mean "the whole team". HierarchicalResourceQuota caps a subtree's total (beta in HNC 1.1). Argo CD's annotation tracking exists partly so HNC copies do not show as drift.
  • Capsule: a cluster-scoped Tenant CRD groups namespaces owned by a user or group; quota, LimitRange, NetworkPolicy, RBAC, allowed StorageClasses and IngressClasses, node selectors and registries defined once on the Tenant are stamped into every namespace the owner creates. Owners create namespaces themselves; the Capsule policy engine (admission) keeps them inside the tenant.
  • vcluster: a tenant cluster whose API server, controller manager and data store (SQLite by default, etcd optional) run in one pod in a host namespace, plus a syncer that copies pods, Services, ConfigMaps and Secrets down to the host namespace where they actually run. Tenants see their own CRDs and cluster-scoped objects; the host sees translated copies. With private nodes a vcluster gets dedicated worker nodes and becomes indistinguishable from a real cluster; with shared nodes it is control-plane isolation only.

Data-plane controls the page lists that your tenant template may lack

  • Storage: dynamic provisioning only, a StorageClass per tenant where it matters, and reclaimPolicy: Delete on shared classes so a released PV cannot be rebound by another namespace.
  • Sandboxing: a RuntimeClass (gVisor, Kata) selected in the pod spec for untrusted code; the container boundary alone is a shared kernel.
  • Node isolation: taint the tenant's pool and let a mutating webhook (or Kyverno mutate rule) add the toleration and nodeAffinity to pods in that tenant's namespaces, so tenants never write scheduling constraints themselves.
  • DNS: CoreDNS answers cross-namespace queries by default; the CoreDNS policy plugin can restrict lookups to the pod's own namespace when the tenancy model requires it.
  • API Priority and Fairness: FlowSchema and PriorityLevelConfiguration protect the API server from a tenant's chatty controller; it matters once tenants run their own operators.
Mental model

Ask what the tenant needs to own: only workloads, then namespaces; CRDs and webhooks too, then a virtual control plane; the kernel and the version skew, then a cluster. Each step up trades idle capacity and operations for a boundary. Naming the boundary the threat model actually requires is the whole answer.

Exercises

tick the dot when its check passes
kubectl -n team-a get resourcequota team-a-quota -o yaml   # read used vs hard
kubectl -n team-a create deploy filler --image=ghcr.io/nginxinc/nginx-unprivileged:1.27-alpine --replicas=25
kubectl -n team-a get rs -l app=filler -o jsonpath='{.items[0].status.replicas}'
kubectl -n team-a get events --sort-by=.lastTimestamp | grep -i quota | tail -3
outputcaptured 2026-08-26
$ kubectl -n team-a get resourcequota team-a-quota -o yaml   # read used vs hard
apiVersion: v1
kind: ResourceQuota
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"v1","kind":"ResourceQuota","metadata":{"annotations":{},"name":"team-a-quota","namespace":"team-a"},"spec":{"hard":{"limits.cpu":"4","limits.memory":"8Gi","persistentvolumeclaims":"4","pods":"20","requests.cpu":"2","requests.memory":"4Gi","services.loadbalancers":"1"}}}
  creationTimestamp: "2026-08-27T02:06:44Z"
  name: team-a-quota
  namespace: team-a
  resourceVersion: "18561"
  uid: 1e256ecb-ab19-4f1f-bce7-0ae59563b845
spec:
  hard:
    limits.cpu: "4"
    limits.memory: 8Gi
    persistentvolumeclaims: "4"
    pods: "20"
    requests.cpu: "2"
    requests.memory: 4Gi
    services.loadbalancers: "1"
status:
  hard:
    limits.cpu: "4"
    limits.memory: 8Gi
    persistentvolumeclaims: "4"
    pods: "20"
    requests.cpu: "2"
    requests.memory: 4Gi
    services.loadbalancers: "1"
  used:
    limits.cpu: "0"
    limits.memory: "0"
    persistentvolumeclaims: "0"
    pods: "0"
    requests.cpu: "0"
    requests.memory: "0"
    services.loadbalancers: "0"
$ kubectl -n team-a create deploy filler --image=ghcr.io/nginxinc/nginx-unprivileged:1.27-alpine --replicas=25
Warning: would violate PodSecurity "restricted:latest": allowPrivilegeEscalation != false (container "nginx-unprivileged" must set securityContext.allowPrivilegeEscalation=false), unrestricted capabilities (container "nginx-unprivileged" must set securityContext.capabilities.drop=["ALL"]), runAsNonRoot != true (pod or container "nginx-unprivileged" must set securityContext.runAsNonRoot=true), seccompProfile (pod or container "nginx-unprivileged" must set securityContext.seccompProfile.type to "RuntimeDefault" or "Localhost")
deployment.apps/filler created
$ kubectl -n team-a get rs -l app=filler -o jsonpath='{.items[0].status.replicas}'
20
$ kubectl -n team-a get events --sort-by=.lastTimestamp | grep -i quota | tail -3
21s         Warning   FailedCreate        replicaset/filler-6d6c49db66   Error creating: pods "filler-6d6c49db66-b9qs8" is forbidden: exceeded quota: team-a-quota, requested: limits.cpu=200m,pods=1, used: limits.cpu=4,pods=20, limited: limits.cpu=4,pods=20
21s         Warning   FailedCreate        replicaset/filler-6d6c49db66   Error creating: pods "filler-6d6c49db66-vlnj4" is forbidden: exceeded quota: team-a-quota, requested: limits.cpu=200m,pods=1, used: limits.cpu=4,pods=20, limited: limits.cpu=4,pods=20
11s         Warning   FailedCreate        replicaset/filler-6d6c49db66   (combined from similar events): Error creating: pods "filler-6d6c49db66-9ktm7" is forbidden: exceeded quota: team-a-quota, requested: limits.cpu=200m,pods=1, used: limits.cpu=4,pods=20, limited: limits.cpu=4,pods=20

Do the arithmetic before peeking: the LimitRange injects 50m requests and 200m limits per container, so requests.cpu: "2" allows 40 pods but limits.cpu: "4" allows exactly 20, tying with pods: "20". Whichever line the event names, you should be able to derive why.

verify: replicas stall below 25, and the ReplicaSet events say exactly which quota line refused. The skill is reading exceeded quota: team-a-quota, requested: ..., used: ..., limited: ... fluently. Clean up with kubectl -n team-a delete deploy filler.

Run a pod with no resources block and read what it got:

kubectl -n team-a run bare --image=busybox:1.37 --restart=Never -- sleep 300
kubectl -n team-a get pod bare -o jsonpath='{.spec.containers[0].resources}' | jq
outputcaptured 2026-08-26
$ kubectl -n team-a run bare --image=busybox:1.37 --restart=Never -- sleep 300
Warning: would violate PodSecurity "restricted:latest": allowPrivilegeEscalation != false (container "bare" must set securityContext.allowPrivilegeEscalation=false), unrestricted capabilities (container "bare" must set securityContext.capabilities.drop=["ALL"]), runAsNonRoot != true (pod or container "bare" must set securityContext.runAsNonRoot=true), seccompProfile (pod or container "bare" must set securityContext.seccompProfile.type to "RuntimeDefault" or "Localhost")
pod/bare created
$ kubectl -n team-a get pod bare -o jsonpath='{.spec.containers[0].resources}' | jq
{
  "limits": {
    "cpu": "200m",
    "memory": "256Mi"
  },
  "requests": {
    "cpu": "50m",
    "memory": "64Mi"
  }
}
verify: requests 50m/64Mi and limits 200m/256Mi, injected by team-a-limits, matching nothing you typed. Then try to exceed the max: set limits.cpu: "2" on a pod and confirm admission rejects it with a LimitRange error, not a quota error.

team-b exists with the same guardrails:

kubectl -n team-b run web --image=ghcr.io/nginxinc/nginx-unprivileged:1.27-alpine --port=8080 --expose
kubectl -n team-a run poke --image=curlimages/curl:8.11.1 --restart=Never -- \
  curl -s -m 5 -o /dev/null -w '%{http_code}' http://web.team-b.svc:8080
kubectl -n team-a wait --for=jsonpath='{.status.phase}'=Failed pod/poke --timeout=60s \
  || kubectl -n team-a get pod poke -o jsonpath='{.status.containerStatuses[0].state.terminated.exitCode}'
outputcaptured 2026-08-26
$ kubectl -n team-b run web --image=ghcr.io/nginxinc/nginx-unprivileged:1.27-alpine --port=8080 --expose
Warning: would violate PodSecurity "restricted:latest": allowPrivilegeEscalation != false (container "web" must set securityContext.allowPrivilegeEscalation=false), unrestricted capabilities (container "web" must set securityContext.capabilities.drop=["ALL"]), runAsNonRoot != true (pod or container "web" must set securityContext.runAsNonRoot=true), seccompProfile (pod or container "web" must set securityContext.seccompProfile.type to "RuntimeDefault" or "Localhost")
service/web created
pod/web created
$ kubectl -n team-a run poke --image=curlimages/curl:8.11.1 --restart=Never -- \
  curl -s -m 5 -o /dev/null -w '%{http_code}' http://web.team-b.svc:8080
Warning: would violate PodSecurity "restricted:latest": allowPrivilegeEscalation != false (container "poke" must set securityContext.allowPrivilegeEscalation=false), unrestricted capabilities (container "poke" must set securityContext.capabilities.drop=["ALL"]), runAsNonRoot != true (pod or container "poke" must set securityContext.runAsNonRoot=true), seccompProfile (pod or container "poke" must set securityContext.seccompProfile.type to "RuntimeDefault" or "Localhost")
pod/poke created
$ kubectl -n team-a wait --for=jsonpath='{.status.phase}'=Failed pod/poke --timeout=60s \
  || kubectl -n team-a get pod poke -o jsonpath='{.status.containerStatuses[0].state.terminated.exitCode}'
pod/poke condition met
verify: the curl times out (exit 28). Then write the pair of policies that would allow exactly this one flow, an egress rule in team-a and an ingress rule in team-b, apply them, and rerun until you get a 200. Both ends must open.

FAULT=netpol make break strips the DNS egress rule. Diagnose it without the answer, but remember drops only show when something tries: the resident nginx pods never resolve anything, so generate the evidence yourself with kubectl -n team-a exec deploy/<whatever runs there> -- nslookup kubernetes.default (or run a busybox pod), then read hubble observe --namespace team-a --verdict DROPPED and see port 53 dying. make break-answer to confirm, make break-fix to restore.

verify: the same nslookup now succeeds.

A scoped quota counts a subset of the namespace rather than all of it, which is how a platform team forbids one class of workload without capping the tenant. The rejection names the quota, so the tenant learns what they hit.

kubectl apply -f - <<'EOF'
apiVersion: v1
kind: ResourceQuota
metadata: { name: no-critical, namespace: team-a }
spec:
  hard: { pods: "0" }
  scopeSelector:
    matchExpressions:
      - scopeName: PriorityClass
        operator: In
        values: [system-cluster-critical]
EOF
kubectl -n team-a run critical --image=nginx:1.27-alpine --restart=Never --overrides='{"spec":{"priorityClassName":"system-cluster-critical","containers":[{"name":"critical","image":"nginx:1.27-alpine","resources":{"requests":{"cpu":"50m","memory":"64Mi"},"limits":{"cpu":"100m","memory":"128Mi"}}}]}}'
kubectl -n team-a run ordinary --image=nginx:1.27-alpine --restart=Never --overrides='{"spec":{"containers":[{"name":"ordinary","image":"nginx:1.27-alpine","resources":{"requests":{"cpu":"50m","memory":"64Mi"},"limits":{"cpu":"100m","memory":"128Mi"}}}]}}'
kubectl -n team-a get pods ordinary
kubectl -n team-a delete pod ordinary --ignore-not-found
kubectl -n team-a delete resourcequota no-critical
outputcaptured 2026-09-13
$ kubectl apply -f - <<'EOF'
apiVersion: v1
kind: ResourceQuota
metadata: { name: no-critical, namespace: team-a }
spec:
  hard: { pods: "0" }
  scopeSelector:
    matchExpressions:
      - scopeName: PriorityClass
        operator: In
        values: [system-cluster-critical]
EOF
resourcequota/no-critical created
$ kubectl -n team-a run critical --image=nginx:1.27-alpine --restart=Never --overrides='{"spec":{"priorityClassName":"system-cluster-critical","containers":[{"name":"critical","image":"nginx:1.27-alpine","resources":{"requests":{"cpu":"50m","memory":"64Mi"},"limits":{"cpu":"100m","memory":"128Mi"}}}]}}'
Warning: would violate PodSecurity "restricted:latest": allowPrivilegeEscalation != false (container "critical" must set securityContext.allowPrivilegeEscalation=false), unrestricted capabilities (container "critical" must set securityContext.capabilities.drop=["ALL"]), runAsNonRoot != true (pod or container "critical" must set securityContext.runAsNonRoot=true), seccompProfile (pod or container "critical" must set securityContext.seccompProfile.type to "RuntimeDefault" or "Localhost")
Error from server (Forbidden): pods "critical" is forbidden: exceeded quota: no-critical, requested: pods=1, used: pods=0, limited: pods=0
$ kubectl -n team-a run ordinary --image=nginx:1.27-alpine --restart=Never --overrides='{"spec":{"containers":[{"name":"ordinary","image":"nginx:1.27-alpine","resources":{"requests":{"cpu":"50m","memory":"64Mi"},"limits":{"cpu":"100m","memory":"128Mi"}}}]}}'
Warning: would violate PodSecurity "restricted:latest": allowPrivilegeEscalation != false (container "ordinary" must set securityContext.allowPrivilegeEscalation=false), unrestricted capabilities (container "ordinary" must set securityContext.capabilities.drop=["ALL"]), runAsNonRoot != true (pod or container "ordinary" must set securityContext.runAsNonRoot=true), seccompProfile (pod or container "ordinary" must set securityContext.seccompProfile.type to "RuntimeDefault" or "Localhost")
pod/ordinary created
$ kubectl -n team-a get pods ordinary
NAME       READY   STATUS    RESTARTS   AGE
ordinary   0/1     Pending   0          0s
$ kubectl -n team-a delete pod ordinary --ignore-not-found
pod "ordinary" deleted from team-a namespace
$ kubectl -n team-a delete resourcequota no-critical
resourcequota "no-critical" deleted from team-a namespace
verify: the critical pod is refused with an "exceeded quota" message naming no-critical, and the ordinary pod is admitted under the tenant's existing quota. One namespace, two answers, decided by a scope.

Quota is not only cpu and memory. count/<resource>.<group> caps any kind the API server knows about, including the CRDs your platform ships, which is the cheapest guardrail against a tenant looping on create.

kubectl create quota objs -n team-a --hard=count/configmaps=3,count/analysistemplates.argoproj.io=1
kubectl -n team-a get quota objs -o jsonpath='{.status}' | jq
kubectl -n team-a create configmap one --from-literal=a=1
kubectl -n team-a create configmap two --from-literal=a=1
kubectl -n team-a create configmap three --from-literal=a=1
kubectl -n team-a get quota objs -o jsonpath='{.status.used}' | jq
kubectl -n team-a delete configmap one two three --ignore-not-found
kubectl -n team-a delete quota objs
outputcaptured 2026-09-13
$ kubectl create quota objs -n team-a --hard=count/configmaps=3,count/analysistemplates.argoproj.io=1
resourcequota/objs created
$ kubectl -n team-a get quota objs -o jsonpath='{.status}' | jq
{
  "hard": {
    "count/analysistemplates.argoproj.io": "1",
    "count/configmaps": "3"
  },
  "used": {
    "count/analysistemplates.argoproj.io": "1",
    "count/configmaps": "2"
  }
}
$ kubectl -n team-a create configmap one --from-literal=a=1
configmap/one created
$ kubectl -n team-a create configmap two --from-literal=a=1
error: failed to create configmap: configmaps "two" is forbidden: exceeded quota: objs, requested: count/configmaps=1, used: count/configmaps=3, limited: count/configmaps=3
$ kubectl -n team-a create configmap three --from-literal=a=1
error: failed to create configmap: configmaps "three" is forbidden: exceeded quota: objs, requested: count/configmaps=1, used: count/configmaps=3, limited: count/configmaps=3
$ kubectl -n team-a get quota objs -o jsonpath='{.status.used}' | jq
{
  "count/analysistemplates.argoproj.io": "1",
  "count/configmaps": "3"
}
$ kubectl -n team-a delete configmap one two three --ignore-not-found
configmap "one" deleted from team-a namespace
$ kubectl -n team-a delete quota objs
resourcequota "objs" deleted from team-a namespace
verify: status.used tracks both keys the moment the quota is created: count/configmaps at 2, because team-a already holds kube-root-ca.crt and one more before you create anything, and count/analysistemplates.argoproj.io at 1 for the success-rate template. So the limit of 3 leaves room for exactly one, one is created and two is refused with a message naming objs and both numbers. Count quotas are charged against what is already there, not against what you add.

The lab does not install HNC, so this block installs it first and removes it at the end. Do it once to see what namespace inheritance actually produces, then decide whether you would run it.

The kubectl hns plugin is not part of the lab's tool set either, so this uses the CRs directly: a SubnamespaceAnchor is what the plugin creates for you.

kubectl apply -f https://github.com/kubernetes-sigs/hierarchical-namespaces/releases/download/v1.1.0/default.yaml
kubectl -n hnc-system rollout status deploy/hnc-controller-manager --timeout=180s
kubectl -n team-a create rolebinding tenant-view --clusterrole=view --user=dev-a
kubectl apply -f - <<'EOF'
apiVersion: hnc.x-k8s.io/v1alpha2
kind: SubnamespaceAnchor
metadata: { name: dev, namespace: team-a }
EOF
sleep 20
kubectl get ns dev
kubectl get hierarchyconfiguration -n dev -o jsonpath='{.items[0].status}' | jq
kubectl get rolebinding -n dev --show-labels
kubectl delete subnamespaceanchor dev -n team-a
kubectl -n team-a delete rolebinding tenant-view
kubectl delete -f https://github.com/kubernetes-sigs/hierarchical-namespaces/releases/download/v1.1.0/default.yaml
outputcaptured 2026-09-13
$ kubectl apply -f https://github.com/kubernetes-sigs/hierarchical-namespaces/releases/download/v1.1.0/default.yaml
namespace/hnc-system created
customresourcedefinition.apiextensions.k8s.io/hierarchicalresourcequotas.hnc.x-k8s.io created
customresourcedefinition.apiextensions.k8s.io/hierarchyconfigurations.hnc.x-k8s.io created
customresourcedefinition.apiextensions.k8s.io/hncconfigurations.hnc.x-k8s.io created
customresourcedefinition.apiextensions.k8s.io/subnamespaceanchors.hnc.x-k8s.io created
role.rbac.authorization.k8s.io/hnc-leader-election-role created
clusterrole.rbac.authorization.k8s.io/hnc-admin-role created
clusterrole.rbac.authorization.k8s.io/hnc-manager-role created
rolebinding.rbac.authorization.k8s.io/hnc-leader-election-rolebinding created
clusterrolebinding.rbac.authorization.k8s.io/hnc-manager-rolebinding created
secret/hnc-webhook-server-cert created
service/hnc-controller-manager-metrics-service created
service/hnc-webhook-service created
deployment.apps/hnc-controller-manager created
mutatingwebhookconfiguration.admissionregistration.k8s.io/hnc-mutating-webhook-configuration created
validatingwebhookconfiguration.admissionregistration.k8s.io/hnc-validating-webhook-configuration created
$ kubectl -n hnc-system rollout status deploy/hnc-controller-manager --timeout=180s
Waiting for deployment "hnc-controller-manager" rollout to finish: 0 of 1 updated replicas are available...
deployment "hnc-controller-manager" successfully rolled out
$ kubectl -n team-a create rolebinding tenant-view --clusterrole=view --user=dev-a
rolebinding.rbac.authorization.k8s.io/tenant-view created
$ kubectl apply -f - <<'EOF'
apiVersion: hnc.x-k8s.io/v1alpha2
kind: SubnamespaceAnchor
metadata: { name: dev, namespace: team-a }
EOF
subnamespaceanchor.hnc.x-k8s.io/dev created
$ sleep 20
$ kubectl get ns dev
NAME   STATUS   AGE
dev    Active   20s
$ kubectl get hierarchyconfiguration -n dev -o jsonpath='{.items[0].status}' | jq
{}
$ kubectl get rolebinding -n dev --show-labels
NAME          ROLE               AGE   LABELS
developer     Role/developer     1s    app.kubernetes.io/managed-by=hnc.x-k8s.io,hnc.x-k8s.io/inherited-from=team-a
pg            Role/pg            1s    app.kubernetes.io/managed-by=hnc.x-k8s.io,cnpg.io/cluster=pg,hnc.x-k8s.io/inherited-from=team-a
tenant-view   ClusterRole/view   1s    app.kubernetes.io/managed-by=hnc.x-k8s.io,hnc.x-k8s.io/inherited-from=team-a
$ kubectl delete subnamespaceanchor dev -n team-a
subnamespaceanchor.hnc.x-k8s.io "dev" deleted from team-a namespace
$ kubectl -n team-a delete rolebinding tenant-view
rolebinding.rbac.authorization.k8s.io "tenant-view" deleted from team-a namespace
$ kubectl delete -f https://github.com/kubernetes-sigs/hierarchical-namespaces/releases/download/v1.1.0/default.yaml
namespace "hnc-system" deleted
customresourcedefinition.apiextensions.k8s.io "hierarchicalresourcequotas.hnc.x-k8s.io" deleted
customresourcedefinition.apiextensions.k8s.io "hierarchyconfigurations.hnc.x-k8s.io" deleted
customresourcedefinition.apiextensions.k8s.io "hncconfigurations.hnc.x-k8s.io" deleted
customresourcedefinition.apiextensions.k8s.io "subnamespaceanchors.hnc.x-k8s.io" deleted
role.rbac.authorization.k8s.io "hnc-leader-election-role" deleted from hnc-system namespace
clusterrole.rbac.authorization.k8s.io "hnc-admin-role" deleted
clusterrole.rbac.authorization.k8s.io "hnc-manager-role" deleted
rolebinding.rbac.authorization.k8s.io "hnc-leader-election-rolebinding" deleted from hnc-system namespace
clusterrolebinding.rbac.authorization.k8s.io "hnc-manager-rolebinding" deleted
secret "hnc-webhook-server-cert" deleted from hnc-system namespace
service "hnc-controller-manager-metrics-service" deleted from hnc-system namespace
service "hnc-webhook-service" deleted from hnc-system namespace
deployment.apps "hnc-controller-manager" deleted from hnc-system namespace
mutatingwebhookconfiguration.admissionregistration.k8s.io "hnc-mutating-webhook-configuration" deleted
validatingwebhookconfiguration.admissionregistration.k8s.io "hnc-validating-webhook-configuration" deleted
verify: the dev namespace appears without anyone creating it, and the RoleBinding you made in team-a is copied into it carrying an hnc.x-k8s.io/inherited-from label. That label is the whole argument for and against the model: propagation you did not ask for, visible in the object.

API Priority and Fairness is already running on every cluster you will be handed, with objects you can read. A tenant hammering the API server is a fairness question before it is a quota question.

kubectl get flowschema
kubectl get prioritylevelconfiguration
kubectl get flowschema -o custom-columns=NAME:.metadata.name,PL:.spec.priorityLevelConfiguration.name,MATCH:.spec.rules[0].subjects[0].kind | head -20
kubectl get --raw /debug/api_priority_and_fairness/dump_priority_levels | head -20
outputcaptured 2026-09-12
$ kubectl get flowschema
NAME                           PRIORITYLEVEL     MATCHINGPRECEDENCE   DISTINGUISHERMETHOD   AGE   MISSINGPL
exempt                         exempt            1                    <none>                47m   False
probes                         exempt            2                    <none>                47m   False
system-leader-election         leader-election   100                  ByUser                47m   False
system-node-high               node-high         400                  ByUser                47m   False
system-nodes                   system            500                  ByUser                47m   False
kube-controller-manager        workload-high     800                  ByNamespace           47m   False
kube-scheduler                 workload-high     800                  ByNamespace           47m   False
kube-system-service-accounts   workload-high     900                  ByNamespace           47m   False
service-accounts               workload-low      9000                 ByUser                47m   False
global-default                 global-default    9900                 ByUser                47m   False
catch-all                      catch-all         10000                ByUser                47m   False
$ kubectl get prioritylevelconfiguration
NAME              TYPE      NOMINALCONCURRENCYSHARES   QUEUES   HANDSIZE   QUEUELENGTHLIMIT   AGE
catch-all         Limited   5                          <none>   <none>     <none>             47m
exempt            Exempt    <none>                     <none>   <none>     <none>             47m
global-default    Limited   20                         128      6          50                 47m
leader-election   Limited   10                         16       4          50                 47m
node-high         Limited   40                         64       6          50                 47m
system            Limited   30                         64       6          50                 47m
workload-high     Limited   40                         128      6          50                 47m
workload-low      Limited   100                        128      6          50                 47m
$ kubectl get flowschema -o custom-columns=NAME:.metadata.name,PL:.spec.priorityLevelConfiguration.name,MATCH:.spec.rules[0].subjects[0].kind | head -20
NAME                           PL                MATCH
catch-all                      catch-all         Group
exempt                         exempt            Group
global-default                 global-default    Group
kube-controller-manager        workload-high     User
kube-scheduler                 workload-high     User
kube-system-service-accounts   workload-high     ServiceAccount
probes                         exempt            Group
service-accounts               workload-low      Group
system-leader-election         leader-election   User
system-node-high               node-high         Group
system-nodes                   system            Group
$ kubectl get --raw /debug/api_priority_and_fairness/dump_priority_levels | head -20
PriorityLevelName, ActiveQueues, IsIdle, IsQuiescing, WaitingRequests, ExecutingRequests, DispatchedRequests, RejectedRequests, TimedoutRequests, CancelledRequests
catch-all,         0,            true,   false,       0,               0,                 59,                 0,                0,                0
exempt,            0,            true,   false,       0,               0,                 6418,               0,                0,                0
global-default,    0,            false,  false,       0,               1,                 4449,               0,                0,                0
leader-election,   0,            true,   false,       0,               0,                 4063,               0,                0,                0
node-high,         0,            true,   false,       0,               0,                 965,                0,                0,                0
system,            0,            true,   false,       0,               0,                 6670,               0,                0,                0
workload-high,     0,            true,   false,       0,               0,                 12303,              0,                0,                0
workload-low,      0,            true,   false,       0,               0,                 48013,              0,                0,                0
verify: you can name the flow schema a request from a tenant ServiceAccount would land in, and the priority level it maps to. The dump endpoint is the one to reach for when the complaint is "the API server is slow for us only".

Self-check

answer before opening
A tenant applies a Deployment. It is accepted, and no pods appear. Where is the error?

One level down, on the ReplicaSet: quota and PSS are pod-level admission decisions, and the Deployment controller only records the failure in the RS's events and conditions. kubectl -n <ns> describe rs -l app=<x> or namespace events. This indirection is a stock exam scenario.

Why does adding a LimitRange sometimes make previously-working pods fail?

Because it enforces min/max as well as injecting defaults. Existing pods keep running (admission is not retroactive), but the next pod whose explicit requests fall outside the bounds is rejected, and the message comes from the LimitRange, not the quota.

Quota says requests.cpu: "2". A tenant asks why they cannot run 40 pods when the LimitRange injects 50m each.

Because the other quota lines bind first: limits.cpu: "4" against a 200m default limit allows 20 pods, and pods: "20" caps it at the same number. Whichever binds first is the one named in the event. The lesson: quota is a conjunction, and the tightest line wins.

Name the isolation ladder and one cost of each rung.

Namespace + quota + policy (cheap, shared kernel and shared API server). Dedicated node pools via taints (no shared kernel; stranded capacity per pool). Virtual clusters (own API server and CRDs; more moving parts to run). Separate clusters (strongest; multiplied operational and idle cost). Pick the thinnest rung that satisfies the actual threat model.

Why is services.loadbalancers: "1" in a tenant quota?

Because a LoadBalancer bills outside the cluster whether or not traffic flows. Quota here is a spend control, not a fairness control: the same reasoning that puts per-StorageClass storage quota in a tenant template.

A tenant needs to install its own CRDs and admission webhooks but you cannot afford a cluster per tenant. Which model?

A virtual control plane per tenant, for example vcluster: the tenant gets its own API server (so CRDs and webhooks are theirs alone) while pods still run on shared nodes through the syncer. Namespaces cannot isolate cluster-scoped objects; a dedicated cluster can, at full duplication cost.

What does HNC add to plain namespaces, in two sentences?

A parent-child hierarchy: RBAC, NetworkPolicies and other chosen objects propagate from a parent to all descendants, and users with rights in a parent can create subnamespaces without cluster-level permission. Tree labels on every namespace let selectors mean "the whole team", and HierarchicalResourceQuota caps a subtree.

Tenants keep launching pods with the PriorityClass reserved for platform components. Fix with a quota.

A ResourceQuota in each tenant namespace with scopeSelector: { matchExpressions: [{ scopeName: PriorityClass, operator: In, values: [platform-critical] }] } and hard: { pods: "0" }. Admission then rejects any pod naming that class in the namespace, with an "exceeded quota" message, while other classes are unaffected.

Why should a StorageClass shared between tenants use reclaimPolicy Delete?

A Retain PV released by one tenant stays in the cluster with its data; PVs are cluster-scoped, so a claim from another namespace could bind to it once the claimRef is cleared. Delete removes the backend volume with the claim, so data cannot cross tenants through the storage layer. Per-tenant classes are the stronger alternative.

Docs to know your way around

study time, not exam time
  • kubernetes.io: Resource Quotas (the full list of countable resources), Limit Ranges, Network Policies.
  • kubernetes.io Concepts → Security → Multi-tenancy: reads like it was written for this competency; the isolation-ladder framing above is theirs.
  • Offline: kubectl explain resourcequota.spec.hard, kubectl explain limitrange.spec.limits, and kubectl describe quota -n <ns> for the used/hard table.
  • kubernetes.io Concepts → Security → Multi-tenancy, "Implementations": namespace-per-tenant versus virtual control plane, and the data-plane list (network, storage, sandboxing, node isolation).
  • github.com/kubernetes-sigs/hierarchical-namespaces (user guide): subnamespaces, propagation, tree labels and HRQ; projectcapsule.dev for the Tenant CRD; vcluster.com/docs architecture page for the syncer and shared vs private nodes.
  • kubernetes.io: Resource Quotas, "Quota scopes" and "Object count quota": the scopeSelector operators and the count/ syntax that also covers custom resources.
free before 1.5make down-sec