5.1 RBAC and secrets: least privilege in practice

RBAC questions are gift questions if the model is exact and time sinks if it is fuzzy. Secrets are the other half of the competency, and the interesting part is not the object; it is the three sanctioned ways to keep secrets out of git.

needsmake up sec

Orientation

competency 5.2 · RBAC and security controls

Every Forbidden error you will ever see (in a workflow node, an operator log, a CI job, an exam task) decomposes the same way: subject, verb, resource, namespace. Learn to read the message and half the domain answers itself.

Authorization in three sentences

A request arrives authenticated as a subject with groups. Every RBAC rule is an allow; there is no deny, and the union of all matching rules is your permission set. If nothing allows it, it is forbidden, which is why debugging RBAC is always "find the missing rule or the missing binding", never "find the rule that blocked me".

The RBAC model, exactly

roles · bindings · subjects · verbs

Object	Scope	Contains
Role	one namespace	rules: apiGroups × resources × verbs (+ resourceNames)
ClusterRole	cluster-wide definition	same, plus cluster-scoped resources and non-resource URLs
RoleBinding	grants in one namespace	subjects + a roleRef to a Role or a ClusterRole
ClusterRoleBinding	grants everywhere	subjects + a roleRef to a ClusterRole

The asymmetry exams love

A RoleBinding can reference a ClusterRole, granting its rules within that one namespace only. That is how you define "developer" once and bind it per tenant. A ClusterRoleBinding grants everywhere and is almost always the wrong default. If a task's third check is "and they must not have this in another namespace", that check exists to catch a ClusterRoleBinding.

Four combinations of role and binding, and where each one actually grants:

Subjects

Users and Groups are asserted by the authentication layer (client certificates, OIDC claims, proxy headers) and are not objects, which is why you can bind to dev-a with no dev-a existing anywhere, and why --as=dev-a works for testing. ServiceAccounts are objects, and they are how software gets identity: system:serviceaccount:<ns>:<name>, in group system:serviceaccounts:<ns>. Every controller failure that says Forbidden traces to some SA's missing rule.

Modern SA tokens are short-lived, audience-bound projected volumes, not the old forever-Secrets, which also makes them the basis of workload identity (section 5.5's SPIFFE IDs are built from the SA name). Setting automountServiceAccountToken: false on pods that never call the API server is a cheap, real hardening step.

Verbs and rules, the details that decide tasks

get, list, watch are separate verbs. A dashboard that lists needs list; a controller needs watch too. Granting get and expecting kubectl get pods (plural) to work is a classic.
Subresources are their own resource strings: pods/exec, pods/log, pods/portforward, deployments/scale, */status. "Can read logs but not exec" is expressible, and is a good tenant default.
resourceNames narrows a rule to named objects: the way to allow editing one ConfigMap. Note it cannot restrict list or watch.
Wildcards exist (*) and least privilege says do not. escalate and bind are the special verbs that let a subject grant permissions they do not themselves hold; treat them as admin-only.
Aggregated ClusterRoles (aggregationRule with label selectors) are how the built-in view/edit/admin roles absorb new CRDs: label your ClusterRole rbac.authorization.k8s.io/aggregate-to-view: "true" and every viewer gains read access to your new kind. That is the platform-engineering move when you ship a CRD.
Built-ins to know: view (read, no secrets), edit (write, no RBAC), admin (edit + manage RBAC in the namespace), cluster-admin (everything).

The two interrogation commands

kubectl auth can-i <verb> <resource> --as=<user> -n <ns>
kubectl auth can-i --list --as=system:serviceaccount:<ns>:<sa> -n <ns>

Every RBAC task should end with one of these as proof. --list is the one that finds surprises: it prints the full effective matrix, including everything inherited from group bindings you forgot about.

Secrets: the controls half

and the three GitOps answers

A stock Secret is base64, not encryption. Anyone with get secrets in the namespace has the plaintext, and so does anyone with etcd access unless encryption-at-rest is configured on the API server (EncryptionConfiguration, optionally backed by a KMS; worth knowing as a phrase). Hence three rules: do not grant get secrets casually, prefer projected/short-lived tokens over long-lived ones, and never let a plain Secret near git.

Approach	What git holds	Who can decrypt	Good for
Sealed Secrets	ciphertext (a SealedSecret CR)	only the in-cluster controller's private key	self-contained clusters, no external store
External Secrets (ESO)	a reference (SecretStore + ExternalSecret)	whoever the store authorises	an existing vault/cloud secret manager is the source of truth
SOPS	partially-encrypted YAML	key holders (age/PGP/KMS)	Flux-native workflows, reviewable diffs

The distinction to be able to state: sealed = encrypted at rest in git; ESO = git holds only pointers, the store holds truth, and rotation happens outside your repo. Both remove plaintext from version control; only ESO gives you central rotation and audit.

ESO's object model, since it is the one you are most likely to meet

SecretStore (namespaced) or ClusterSecretStore (cluster-wide, referenced with kind: ClusterSecretStore) declares where the secrets live and how to authenticate: a provider block (vault, aws, gcp, azure, kubernetes, fake for practice) plus credentials, usually a ServiceAccount token or a Secret reference.
ExternalSecret declares what to materialise: secretStoreRef, a refreshInterval, and either explicit data[] entries (remote key → local key) or dataFrom to pull a whole path. target.name names the Secret it creates, target.creationPolicy decides whether ESO owns it, and target.template lets you assemble a shaped Secret (a full config file, a connection string) instead of raw key-value pairs.
Status conditions to read: SecretSynced for success; a failure names the store, the missing key, or the auth error, in that order of likelihood.
PushSecret goes the other way, for the rare case where the cluster is the source of truth.

The exam-shaped skill is the same one as everywhere else in this domain: read the CRD, wire two objects together, then prove it with the materialised Secret rather than the apply exit code.

Sealed Secrets' sharp edge

The sealed value is encrypted for a specific controller and (by default) a specific namespace/name. Rebuild the cluster without restoring the controller's key and every SealedSecret in git becomes undecryptable, so the sealing key is a backup item. Knowing that is the difference between recommending it and recommending it responsibly.

Exercises

tick the dot when its check passes

examples/multitenancy/team-a.yaml binds Role developer to user dev-a. Predict, then check, each of these:

kubectl auth can-i create deployments --as=dev-a -n team-a
kubectl auth can-i delete secrets     --as=dev-a -n team-a
kubectl auth can-i list secrets       --as=dev-a -n team-a
kubectl auth can-i create pods        --as=dev-a -n team-b
kubectl auth can-i list nodes         --as=dev-a

verify: yes, no, yes, no, no, and for each "no", name the missing piece (rule vs binding vs scope). The secrets split (read yes, write no) is deliberate least-privilege design; find the comment in the file.

Convert developer into a ClusterRole and bind it into team-b with a RoleBinding for user dev-b.

verify: kubectl auth can-i create deployments --as=dev-b -n team-b yes, -n team-a no. One definition, per-tenant grants; this shape is the answer to most "design RBAC for tenants" prompts.

Create SA reporter in team-a allowed only to get,list pods, then prove it from inside:

kubectl -n team-a create sa reporter
kubectl -n team-a create role pod-reader --verb=get,list --resource=pods
kubectl -n team-a create rolebinding reporter --role=pod-reader --serviceaccount=team-a:reporter
kubectl -n team-a run api-probe --image=bitnami/kubectl:latest --restart=Never \
  --overrides='{"spec":{"serviceAccountName":"reporter"}}' -- get pods
kubectl -n team-a logs api-probe

(The image's entrypoint is already kubectl, so the args are just get pods; doubling it up runs kubectl kubectl.)

verify: the pod lists pods successfully; then re-run with -- get secrets and read the Forbidden message, noting it names the SA, the verb, and the resource. That message format is the same one you met in workflow failures (3.4) and will meet again in operator logs.

The controller from make sec is in kube-system:

kubectl -n team-a create secret generic db-pass --from-literal=password=hunter2 \
  --dry-run=client -o yaml | kubeseal --controller-namespace kube-system --controller-name sealed-secrets -o yaml > sealed.yaml
grep -c hunter2 sealed.yaml   # must print 0
kubectl apply -f sealed.yaml
kubectl -n team-a get secret db-pass -o jsonpath='{.data.password}' | base64 -d

verify: the sealed file contains no plaintext, yet the unsealed Secret round-trips to hunter2. Now delete the Secret only and watch the controller recreate it from the SealedSecret; that reconcile is why the sealed form is the source of truth you commit.

External Secrets ships a fake provider made for exactly this practice: create a SecretStore of provider fake holding key pg/password, an ExternalSecret targeting it, and verify the materialised Secret appears with the value, refreshed on interval.

verify: kubectl get externalsecret shows SecretSynced True. The provider is fake; the CRD mechanics, which are what the exam could touch, are entirely real.

FAULT=rbac make break. Everything looks Running; only the can-i matrix shows the hole.

verify: found and fixed in 7 minutes, proven by the restored auth can-i output.

Self-check

answer before opening

Define "developer" once and grant it in twelve namespaces. What objects, and how many?

One ClusterRole plus twelve RoleBindings (each referencing that ClusterRole, each in its namespace). Not a ClusterRoleBinding: that would grant everywhere and fail the "and not in the other namespace" check that graders love.

A user can get a pod by name but kubectl get pods fails. Why?

list is a separate verb from get, and a rule with resourceNames cannot grant list at all. Add list (and watch if anything streams), accepting that list exposes every object of that kind in the namespace.

How do you let a tenant read pod logs but never exec into a pod?

Grant get on pods/log and do not grant create on pods/exec. Subresources are distinct resource strings, which is what makes this expressible at all.

You ship a new CRD. How do existing view users get read access without editing the built-in role?

Create a ClusterRole with read verbs on your kind and label it rbac.authorization.k8s.io/aggregate-to-view: "true". Aggregation folds it into the built-in role automatically, the standard platform-team move when adding an API.

Sealed Secrets versus ESO: which would you pick for a regulated environment with an existing vault, and why?

ESO: the vault stays the single source of truth with its own audit and rotation, and git holds only references; nothing secret is ever committed, encrypted or not. Sealed Secrets suits clusters with no external store, at the cost of managing (and backing up) the sealing key yourself.

Is a Kubernetes Secret encrypted?

Not by itself: base64 is encoding. At rest it is only encrypted if the API server is configured with an EncryptionConfiguration (optionally KMS-backed); in transit it is protected by TLS. Access control (who has get secrets) is doing most of the actual work.

Docs to know your way around

study time, not exam time

kubernetes.io: Using RBAC Authorization (the RoleBinding-to-ClusterRole pattern and aggregation are both spelled out there); Managing Service Accounts; Encrypting Secret Data at Rest.
sealed-secrets and external-secrets docs: one page each on their CRDs; the fake provider is under ESO's provider list.
Offline: kubectl auth can-i --list, kubectl api-resources --verbs=list, kubectl create role --help (its examples are a rules cheat sheet).