5.1 RBAC and secrets: least privilege in practice

RBAC questions are gift questions if the model is exact and time sinks if it is fuzzy. Secrets are the other half of the competency. The interesting part is not the object but the three sanctioned ways to keep secrets out of git.

needsmake up secmake gitops

Orientation

competency 5.2 · RBAC and security controls

Every Forbidden error you will ever see (in a workflow node, an operator log, a CI job, an exam task) decomposes the same way: subject, verb, resource, namespace. Learn to read the message and half the domain answers itself.

Authorization in three sentences

A request arrives authenticated as a subject with groups. Every RBAC rule is an allow; there is no deny. Your permission set is the union of all matching rules. If nothing allows it, it is forbidden, which is why debugging RBAC is always "find the missing rule or the missing binding", never "find the rule that blocked me".

The RBAC model

roles · bindings · subjects · verbs

Object	Scope	Contains
Role	one namespace	rules: apiGroups × resources × verbs (+ resourceNames)
ClusterRole	cluster-wide definition	same, plus cluster-scoped resources and non-resource URLs
RoleBinding	grants in one namespace	subjects + a roleRef to a Role or a ClusterRole
ClusterRoleBinding	grants everywhere	subjects + a roleRef to a ClusterRole

The asymmetry exams love

A RoleBinding can reference a ClusterRole, granting its rules within that one namespace only. That is how you define "developer" once and bind it per tenant. A ClusterRoleBinding grants everywhere and is almost always the wrong default. If a task's third check is "and they must not have this in another namespace", that check exists to catch a ClusterRoleBinding.

Four combinations of role and binding, and where each one actually grants:

Subjects

Users and Groups are asserted by the authentication layer (client certificates, OIDC claims, proxy headers) and are not objects, which is why you can bind to dev-a with no dev-a existing anywhere, and why --as=dev-a works for testing. ServiceAccounts are objects, and they are how software gets identity: system:serviceaccount:<ns>:<name>, in group system:serviceaccounts:<ns>. Every controller failure that says Forbidden traces to some SA's missing rule.

Modern SA tokens are short-lived, audience-bound, and delivered by projected volume, not the old forever-Secrets, which also makes them the basis of workload identity (section 5.5's SPIFFE IDs are built from the SA name). Setting automountServiceAccountToken: false on pods that never call the API server is a cheap, real hardening step.

Verbs and rules, the details that decide tasks

get, list, watch are separate verbs. A dashboard that lists needs list; a controller needs watch too. Granting get and expecting kubectl get pods (plural) to work is a classic mistake.
Subresources are their own resource strings: pods/exec, pods/log, pods/portforward, deployments/scale, */status. "Can read logs but not exec" is expressible, and is a good tenant default.
resourceNames narrows a rule to named objects: the way to allow editing one ConfigMap. Note it cannot restrict create or deletecollection, and a plain list or watch only passes with a matching metadata.name field selector.
Wildcards (*) exist; least privilege says not to use them. escalate and bind are the special verbs that let a subject grant permissions they do not themselves hold; treat them as admin-only.
Aggregated ClusterRoles (aggregationRule with label selectors) are how the built-in view/edit/admin roles absorb new CRDs: label your ClusterRole rbac.authorization.k8s.io/aggregate-to-view: "true" and every viewer gains read access to your new kind. That is the platform-engineering move when you ship a CRD.
Built-ins to know: view (read, no secrets), edit (write, no RBAC), admin (edit + manage RBAC in the namespace), cluster-admin (everything).

The two interrogation commands

kubectl auth can-i <verb> <resource> --as=<user> -n <ns>
kubectl auth can-i --list --as=system:serviceaccount:<ns>:<sa> -n <ns>

outputcaptured 2026-08-26

$ kubectl auth can-i create deployments --as=dev-a -n team-a
yes
$ kubectl auth can-i --list --as=system:serviceaccount:team-a:reporter -n team-a
Resources                                       Non-Resource URLs                      Resource Names   Verbs
selfsubjectreviews.authentication.k8s.io        []                                     []               [create]
selfsubjectaccessreviews.authorization.k8s.io   []                                     []               [create]
selfsubjectrulesreviews.authorization.k8s.io    []                                     []               [create]
pods                                            []                                     []               [get list]
                                                [/.well-known/openid-configuration/]   []               [get]
                                                [/.well-known/openid-configuration]    []               [get]
                                                [/api/*]                               []               [get]
                                                [/api]                                 []               [get]
                                                [/apis/*]                              []               [get]
                                                [/apis]                                []               [get]
                                                [/healthz]                             []               [get]
                                                [/healthz]                             []               [get]
                                                [/livez]                               []               [get]
                                                [/livez]                               []               [get]
                                                [/openapi/*]                           []               [get]
                                                [/openapi]                             []               [get]
                                                [/openid/v1/jwks/]                     []               [get]
                                                [/openid/v1/jwks]                      []               [get]
                                                [/readyz]                              []               [get]
                                                [/readyz]                              []               [get]
                                                [/version/]                            []               [get]
                                                [/version/]                            []               [get]
                                                [/version]                             []               [get]
                                                [/version]                             []               [get]

Every RBAC task should end with one of these as proof. --list is the one that finds surprises: it prints the full effective matrix, including everything inherited from group bindings you forgot about.

RBAC edges that decide tasks

special verbs, identity checks, tokens, OIDC groups

Verbs and rules people get wrong

Escalation prevention. You can create or update a Role only if you already hold every permission it grants, or hold the escalate verb on roles/clusterroles. You can create a binding only if you hold the permissions being bound, or hold the bind verb on the role (optionally narrowed with resourceNames). The error is attempt to grant extra privileges. A tenant admin who "cannot bind the view role" needs bind on that one ClusterRole, not cluster-admin.
impersonate on users, groups, serviceaccounts (and userextras) is what kubectl --as and --as-group need. It is granted to cluster-admin and to nobody else by default; a tenant testing their own SA with --as needs it explicitly.
kubectl auth whoami (the SelfSubjectReview API) prints the username, uid, groups and extra attributes the API server sees after authentication. It is the first command when an OIDC user or a workload gets Forbidden and you are not sure which identity arrived: the prefix, the group names and the impersonation all show up here.
nonResourceURLs (/metrics, /healthz, /logs/*) exist only in ClusterRoles; a scraper hitting the API server's /metrics needs get on that URL, and the wildcard is a suffix glob.
Default ClusterRoles. view reads most namespaced objects but not Secrets, Roles or RoleBindings. edit adds writes and, less obviously, can read and write Secrets and run pods as any ServiceAccount in the namespace, which is why it is often too much for a developer. admin adds Role and RoleBinding management inside the namespace; cluster-admin is everything. All four aggregate labeled roles (rbac.authorization.k8s.io/aggregate-to-view|edit|admin). system:authenticated holds system:basic-user and system:discovery; if system:unauthenticated can read discovery, that is the system:public-info-viewer binding.
Auto-reconciliation. Default roles and bindings are re-created at API server start unless annotated rbac.authorization.kubernetes.io/autoupdate: "false"; edits to view itself are overwritten, which is why aggregation exists. kubectl auth reconcile -f rbac.yaml applies RBAC manifests additively and reports what changed.
kubectl create role forms. --verb=get,list --resource=pods,pods/log, --resource=deployments.apps for a group-qualified resource, --resource-name=x for named objects, and kubectl create clusterrole --aggregation-rule=.... Generating and then reading the YAML beats writing it from memory.

Identity mechanics

Tokens. Pods get a projected token (serviceAccountToken source with expirationSeconds and audience) refreshed by the kubelet through the TokenRequest API; it is bound to the pod and dies with it. kubectl create token <sa> --duration=1h mints one for scripts; --bound-object-kind Pod --bound-object-name x binds it to an object. Secret-based long-lived tokens are not auto-created since 1.24 and are cleaned up when unused; a task that says "the CI system needs a token" wants kubectl create token or an explicit Secret of type kubernetes.io/service-account-token only if it truly needs a non-expiring one. automountServiceAccountToken: false on the pod or the SA removes the volume.
OIDC. Legacy flags: --oidc-issuer-url, --oidc-client-id, --oidc-username-claim (default sub), --oidc-username-prefix, --oidc-groups-claim, --oidc-groups-prefix. The structured form is an AuthenticationConfiguration file passed with --authentication-config: jwt[].issuer.url/audiences, claimMappings.username.claim|prefix, claimMappings.groups.claim|prefix, CEL claimValidationRules, several issuers. Groups arrive with the prefix (oidc:platform-team), so a RoleBinding to platform-team without the prefix silently matches nobody; kubectl auth whoami shows the real string.
Groups you get for free. system:serviceaccounts (all SAs), system:serviceaccounts:<ns> (one namespace), system:masters (bypasses RBAC entirely; never bind a real user into it), system:nodes.

How this gets tested

Three variants: "user X must be able to do Y in namespace Z and nothing else", "the CI ServiceAccount must be able to submit workflows and read their status", and "explain why user X gets Forbidden although a binding exists" (prefixes, group names, the binding referencing a Role in the wrong namespace, or resourceNames on a list). End every one with kubectl auth can-i --as and, for real identities, kubectl auth whoami.

Secrets: the controls half

and the three GitOps answers

A stock Secret is base64, not encryption. Anyone with get secrets in the namespace has the plaintext. So does anyone with etcd access, unless encryption-at-rest is configured on the API server (EncryptionConfiguration, optionally backed by a KMS; worth knowing as a phrase). Hence three rules: do not grant get secrets casually, prefer projected/short-lived tokens over long-lived ones, and never let a plain Secret near git.

Approach	What git holds	Who can decrypt	Good for
Sealed Secrets	ciphertext (a SealedSecret CR)	only the in-cluster controller's private key	self-contained clusters, no external store
External Secrets (ESO)	a reference (SecretStore + ExternalSecret)	whoever the store authorizes	an existing vault/cloud secret manager is the source of truth
SOPS	partially-encrypted YAML	key holders (age/PGP/KMS)	Flux-native workflows, reviewable diffs

The distinction to be able to state: sealed = encrypted at rest in git; ESO = git holds only pointers, the store holds truth, and rotation happens outside your repo. Both remove plaintext from version control; only ESO gives you central rotation and audit.

ESO's object model, since it is the one you are most likely to meet

SecretStore (namespaced) or ClusterSecretStore (cluster-wide, referenced with kind: ClusterSecretStore) declares where the secrets live and how to authenticate: a provider block (vault, aws, gcp, azure, kubernetes, fake for practice) plus credentials, usually a ServiceAccount token or a Secret reference.
ExternalSecret declares what to materialize: secretStoreRef, a refreshInterval, and either explicit data[] entries (remote key → local key) or dataFrom to pull a whole path. target.name names the Secret it creates, target.creationPolicy decides whether ESO owns it, and target.template lets you assemble a purpose-built Secret (a full config file, a connection string) instead of raw key-value pairs.
Status conditions to read: SecretSynced for success; a failure names the store, the missing key, or the auth error, in that order of likelihood.
PushSecret goes the other way, for the rare case where the cluster is the source of truth.

The skill the exam tests is the same one as everywhere else in this domain: read the CRD, wire two objects together, then prove it with the materialized Secret rather than the apply exit code.

Sealed Secrets' sharp edge

The sealed value is encrypted for a specific controller and (by default) a specific namespace/name. Rebuild the cluster without restoring the controller's key and every SealedSecret in git becomes undecryptable, so the sealing key is a backup item.

Secrets: the fields and formats

encryption at rest, ESO, Sealed Secrets, SOPS, CSI

Encryption at rest

An EncryptionConfiguration (apiserver.config.k8s.io/v1) passed with --encryption-provider-config lists resources[], each with the resource names to cover (secrets, configmaps, custom resources such as externalsecrets.external-secrets.io, or wildcards like *.*) and an ordered providers[]. The first provider encrypts new writes; all listed providers can decrypt, which is how rotation works: add the new key first, restart, then kubectl get secrets -A -o json | kubectl replace -f - to rewrite every object, then remove the old key. Providers: identity (no encryption; the default if the file is absent), aescbc, aesgcm (needs rotation every 200k writes), secretbox, and kms with apiVersion: v2 for an external KMS plugin over a Unix socket (v1 is deprecated). Wildcards need 1.27+, custom resources 1.26+. Reading etcd directly shows k8s:enc:aescbc:v1:key1: prefixes when it is working.

External Secrets Operator, the decisive fields

apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata: { name: db, namespace: team-a }
spec:
  secretStoreRef: { name: vault, kind: ClusterSecretStore }
  refreshPolicy: Periodic            # CreatedOnce | Periodic (default) | OnChange
  refreshInterval: 1h                # 0 means create once, never update
  target:
    name: db-credentials
    creationPolicy: Owner            # Owner | Orphan | Merge | None
    deletionPolicy: Retain           # Retain | Delete | Merge
    template:
      engineVersion: v2
      data:
        DSN: "postgres://{{ .username }}:{{ .password }}@db:5432/app"
  data:
    - secretKey: username
      remoteRef: { key: team-a/db, property: username }
  dataFrom:
    - extract: { key: team-a/db }    # every property becomes a key

refreshPolicy: Periodic syncs every refreshInterval and on spec change; OnChange only when the ExternalSecret changes; CreatedOnce once per object (recreating the ExternalSecret re-fetches, so pair it with target.immutable: true for credentials nothing may overwrite). kubectl annotate es x force-sync=$(date +%s) triggers an immediate sync.
creationPolicy: Owner (ESO owns the Secret, deletes it with the ExternalSecret), Orphan (leaves it), Merge (adds keys to an existing Secret you own), None (never creates). deletionPolicy decides what happens to the Secret when the remote key disappears.
dataFrom.extract pulls a whole JSON object as keys; dataFrom.find selects by path regex or tags; rewrite renames keys. template (engine v2, Go templates with Sprig) builds config files and connection strings; templateFrom reads the template out of a ConfigMap; mergePolicy: Merge keeps the raw keys alongside the templated ones.
Scoping: SecretStore is namespaced; ClusterSecretStore can restrict who may use it with conditions[].namespaceSelector or namespaces. ClusterExternalSecret stamps one ExternalSecret into every namespace matching namespaceSelectors. PushSecret reverses the flow (cluster Secret to store) with updatePolicy: Replace|IfNotExists.
Status: condition Ready with reason SecretSynced or SecretSyncedError and a message naming the store, the missing key or the auth error, in that order of likelihood. The store has its own Ready condition (Valid reason) that fails first when credentials are wrong.

The other three tools, in fields

Sealed Secrets. Scope decides where a ciphertext may be unsealed: strict (default; name and namespace are part of the encryption, so renaming gives decryption error), namespace-wide (annotation sealedsecrets.bitnami.com/namespace-wide: "true", any name in that namespace), cluster-wide (any namespace). kubeseal --scope, --fetch-cert to seal offline, --merge-into to add one key, --raw for a single value. The controller rotates its sealing key every 30 days and keeps old ones, so old ciphertexts still open; back up the sealed-secrets-key* Secrets in kube-system. Annotating an existing Secret sealedsecrets.bitnami.com/managed: "true" lets a SealedSecret take it over; .../patch: "true" merges instead of replacing.
SOPS with Flux. A .sops.yaml with creation_rules (path_regex, age or pgp or kms recipients, encrypted_regex: ^(data|stringData)$ so metadata stays readable) and sops --encrypt --in-place secret.yaml. The Kustomization declares spec.decryption: { provider: sops, secretRef: { name: sops-age } }, where the Secret holds the age key file (age.agekey) or a GPG private key; kustomize-controller decrypts in memory before applying. Diffs stay reviewable because only values are ciphertext.
Secrets Store CSI driver. A SecretProviderClass (provider vault, azure, aws, gcp) plus a pod volume csi.driver: secrets-store.csi.k8s.io with volumeAttributes.secretProviderClass; values are mounted as files, never stored in etcd unless secretObjects asks the driver to sync them into a Kubernetes Secret (for env vars). Rotation is --enable-secret-rotation and --rotation-poll-interval on the driver. Pick it when "the secret must never exist in etcd"; pick ESO when workloads already read Kubernetes Secrets.

Trap

list and watch on Secrets return the values, not just the names. A "read-only" role that includes them has full access to every secret in the namespace. The Kubernetes good-practices page says to grant get on named Secrets (resourceNames) and reserve list/watch for controllers that need them; that sentence is a defensible exam answer on its own.

Exercises

tick the dot when its check passes

examples/multitenancy/team-a.yaml binds Role developer to user dev-a. Predict, then check, each of these:

kubectl auth can-i create deployments --as=dev-a -n team-a
kubectl auth can-i delete secrets     --as=dev-a -n team-a
kubectl auth can-i list secrets       --as=dev-a -n team-a
kubectl auth can-i create pods        --as=dev-a -n team-b
kubectl auth can-i list nodes         --as=dev-a

outputcaptured 2026-08-26

$ kubectl auth can-i create deployments --as=dev-a -n team-a
yes
$ kubectl auth can-i delete secrets     --as=dev-a -n team-a
no
$ kubectl auth can-i list secrets       --as=dev-a -n team-a
yes
$ kubectl auth can-i create pods        --as=dev-a -n team-b
no
$ kubectl auth can-i list nodes         --as=dev-a
Warning: resource 'nodes' is not namespace scoped

no

verify: yes, no, yes, no, no, and for each "no", name the missing piece (rule vs binding vs scope). The secrets split (read yes, write no) is deliberate least-privilege design; find the comment in the file.

Convert developer into a ClusterRole and bind it into team-b with a RoleBinding for user dev-b.

verify: kubectl auth can-i create deployments --as=dev-b -n team-b yes, -n team-a no. One definition, per-tenant grants; this pattern is the answer to most "design RBAC for tenants" prompts.

Create SA reporter in team-a allowed only to get,list pods, then prove it from inside. One obstacle first: team-a's deny-all egress also blocks the API server (you met this in 3.3), so open that path by applying the allow-apiserver CiliumNetworkPolicy from the comments in examples/crossplane/pg-cluster.yaml, and delete it when done:

kubectl -n team-a create sa reporter
kubectl -n team-a create role pod-reader --verb=get,list --resource=pods
kubectl -n team-a create rolebinding reporter --role=pod-reader --serviceaccount=team-a:reporter
kubectl -n team-a run api-probe --image=bitnami/kubectl:latest --restart=Never \
  --overrides='{"spec":{"serviceAccountName":"reporter"}}' -- get pods
kubectl -n team-a logs api-probe

outputcaptured 2026-08-26

$ kubectl -n team-a create sa reporter
serviceaccount/reporter created
$ kubectl -n team-a create role pod-reader --verb=get,list --resource=pods
role.rbac.authorization.k8s.io/pod-reader created
$ kubectl -n team-a create rolebinding reporter --role=pod-reader --serviceaccount=team-a:reporter
rolebinding.rbac.authorization.k8s.io/reporter created
$ kubectl -n team-a run api-probe --image=bitnami/kubectl:latest --restart=Never \
  --overrides='{"spec":{"serviceAccountName":"reporter"}}' -- get pods
Warning: would violate PodSecurity "restricted:latest": allowPrivilegeEscalation != false (container "api-probe" must set securityContext.allowPrivilegeEscalation=false), unrestricted capabilities (container "api-probe" must set securityContext.capabilities.drop=["ALL"]), runAsNonRoot != true (pod or container "api-probe" must set securityContext.runAsNonRoot=true), seccompProfile (pod or container "api-probe" must set securityContext.seccompProfile.type to "RuntimeDefault" or "Localhost")
pod/api-probe created
$ kubectl -n team-a logs api-probe
NAME                            READY   STATUS      RESTARTS   AGE
api-probe                       1/1     Running     0          13s
chatty                          0/1     Completed   0          5m8s
demo-5cfc49f565-6zqmt           1/1     Running     0          97s
demo-5cfc49f565-nx4r8           1/1     Running     0          7m34s
demo-5cfc49f565-trpgb           1/1     Running     0          7m34s
demo-5cfc49f565-w7c85           1/1     Running     0          97s
staging-demo-66955f8974-2xxd2   1/1     Running     0          26m
staging-demo-66955f8974-pzxdh   1/1     Running     0          26m
staging-demo-66955f8974-tzpqx   1/1     Running     0          26m

(The image's entrypoint is already kubectl, so the args are just get pods; doubling it up runs kubectl kubectl.)

verify: the pod lists pods successfully; then re-run with -- get secrets and read the Forbidden message, noting it names the SA, the verb, and the resource. That message format is the same one you met in workflow failures (3.4) and will meet again in operator logs.

The controller from make sec is in kube-system:

kubectl -n team-a create secret generic db-pass --from-literal=password=hunter2 \
  --dry-run=client -o yaml | kubeseal --controller-namespace kube-system --controller-name sealed-secrets -o yaml > sealed.yaml
grep -c hunter2 sealed.yaml   # must print 0
kubectl apply -f sealed.yaml
kubectl -n team-a get secret db-pass -o jsonpath='{.data.password}' | base64 -d

outputcaptured 2026-08-26

$ kubectl -n team-a create secret generic db-pass --from-literal=password=hunter2 \
  --dry-run=client -o yaml | kubeseal --controller-namespace kube-system --controller-name sealed-secrets -o yaml > sealed.yaml
$ grep -c hunter2 sealed.yaml   # must print 0
0
$ kubectl apply -f sealed.yaml
sealedsecret.bitnami.com/db-pass created
$ kubectl -n team-a get secret db-pass -o jsonpath='{.data.password}' | base64 -d
hunter2

verify: the sealed file contains no plaintext, yet the unsealed Secret round-trips to hunter2. Now delete the Secret only and watch the controller recreate it from the SealedSecret; that reconcile is why the sealed form is the source of truth you commit.

External Secrets ships a fake provider made for exactly this practice: create a SecretStore of provider fake holding key pg/password, an ExternalSecret targeting it, and verify the materialized Secret appears with the value, refreshed on interval.

verify: kubectl get externalsecret shows SecretSynced True. The provider is fake; the CRD mechanics, which are what the exam could touch, are entirely real.

FAULT=rbac make break. Everything looks Running; only the can-i matrix shows the hole.

verify: found and fixed in 7 minutes, proven by the restored auth can-i output.

kubectl auth whoami answers the question every impersonation task starts with. Run it under --as and you can see exactly what the API server will evaluate rules against, including the groups you may have forgotten to pass.

kubectl auth whoami
kubectl auth whoami --as dev-a
kubectl auth whoami --as dev-a --as-group platform --as-group system:authenticated
kubectl auth can-i --list --as dev-a --as-group platform -n team-a | head -15

outputcaptured 2026-09-12

$ kubectl auth whoami
ATTRIBUTE                                           VALUE
Username                                            kubernetes-admin
Groups                                              [kubeadm:cluster-admins system:authenticated]
Extra: authentication.kubernetes.io/credential-id   [X509SHA256=5e1514c2eb0eb304bb87c1a2eb03282a5074648ec504dd0ed919195f9d9017d5]
$ kubectl auth whoami --as dev-a
ATTRIBUTE   VALUE
Username    dev-a
Groups      [system:authenticated]
$ kubectl auth whoami --as dev-a --as-group platform --as-group system:authenticated
ATTRIBUTE   VALUE
Username    dev-a
Groups      [platform system:authenticated]
$ kubectl auth can-i --list --as dev-a --as-group platform -n team-a | head -15
Resources                                       Non-Resource URLs   Resource Names   Verbs
selfsubjectreviews.authentication.k8s.io        []                  []               [create]
selfsubjectaccessreviews.authorization.k8s.io   []                  []               [create]
selfsubjectrulesreviews.authorization.k8s.io    []                  []               [create]
configmaps                                      []                  []               [get list watch create update patch delete]
deployments                                     []                  []               [get list watch create update patch delete]
jobs                                            []                  []               [get list watch create update patch delete]
pods/log                                        []                  []               [get list watch create update patch delete]
pods                                            []                  []               [get list watch create update patch delete]
services                                        []                  []               [get list watch create update patch delete]
statefulsets                                    []                  []               [get list watch create update patch delete]
configmaps.apps                                 []                  []               [get list watch create update patch delete]
deployments.apps                                []                  []               [get list watch create update patch delete]
jobs.apps                                       []                  []               [get list watch create update patch delete]
pods.apps/log                                   []                  []               [get list watch create update patch delete]

verify: the username and the group list come back exactly as you passed them. Note that impersonation without --as-group drops you into only the default groups, which is why a binding to a group appears not to work.

RBAC forbids privilege escalation: you cannot create a binding that grants permissions you do not hold. The escape is the bind verb on the specific role, and the error before it is one you should recognize instantly.

kubectl -n team-a create rolebinding dev-a-admin --clusterrole=admin --user=dev-a
kubectl create clusterrole node-reader --verb=get,list --resource=nodes
kubectl -n team-a create rolebinding node-reader-binding --clusterrole=node-reader --user=dev-b --as dev-a
kubectl create clusterrole binder --verb=bind --resource=clusterroles --resource-name=node-reader
kubectl -n team-a create rolebinding dev-a-binder --clusterrole=binder --user=dev-a
kubectl -n team-a create rolebinding node-reader-binding --clusterrole=node-reader --user=dev-b --as dev-a
kubectl -n team-a delete rolebinding node-reader-binding dev-a-binder dev-a-admin --ignore-not-found
kubectl delete clusterrole node-reader binder

outputcaptured 2026-09-12

$ kubectl -n team-a create rolebinding dev-a-admin --clusterrole=admin --user=dev-a
rolebinding.rbac.authorization.k8s.io/dev-a-admin created
$ kubectl create clusterrole node-reader --verb=get,list --resource=nodes
clusterrole.rbac.authorization.k8s.io/node-reader created
$ kubectl -n team-a create rolebinding node-reader-binding --clusterrole=node-reader --user=dev-b --as dev-a
error: failed to create rolebinding: rolebindings.rbac.authorization.k8s.io "node-reader-binding" is forbidden: user "dev-a" (groups=["system:authenticated"]) is attempting to grant RBAC permissions not currently held:
{APIGroups:[""], Resources:["nodes"], Verbs:["get" "list"]}
$ kubectl create clusterrole binder --verb=bind --resource=clusterroles --resource-name=node-reader
clusterrole.rbac.authorization.k8s.io/binder created
$ kubectl -n team-a create rolebinding dev-a-binder --clusterrole=binder --user=dev-a
rolebinding.rbac.authorization.k8s.io/dev-a-binder created
$ kubectl -n team-a create rolebinding node-reader-binding --clusterrole=node-reader --user=dev-b --as dev-a
rolebinding.rbac.authorization.k8s.io/node-reader-binding created
$ kubectl -n team-a delete rolebinding node-reader-binding dev-a-binder dev-a-admin --ignore-not-found
rolebinding.rbac.authorization.k8s.io "node-reader-binding" deleted from team-a namespace
rolebinding.rbac.authorization.k8s.io "dev-a-binder" deleted from team-a namespace
rolebinding.rbac.authorization.k8s.io "dev-a-admin" deleted from team-a namespace
$ kubectl delete clusterrole node-reader binder
clusterrole.rbac.authorization.k8s.io "node-reader" deleted
clusterrole.rbac.authorization.k8s.io "binder" deleted

verify: the first attempt fails with "attempt to grant extra privileges" listing the rules you lack, and it succeeds once bind on that exact ClusterRole is granted. Say why resourceNames on the bind rule matters more than the verb.

A ServiceAccount token is a JWT with an expiry and an audience, and since it stopped being a Secret by default, knowing how to mint and inspect one is basic hygiene. The bound claims are what make it safe.

kubectl -n team-a create serviceaccount reporter --dry-run=client -o yaml | kubectl apply -f -
TOKEN=$(kubectl -n team-a create token reporter --duration=10m)
echo "$TOKEN" | cut -d. -f2 | tr '_-' '/+' | base64 -d 2>/dev/null | jq
kubectl -n team-a run tokenholder --image=nginx:1.27-alpine --restart=Never --overrides='{"spec":{"serviceAccountName":"reporter","containers":[{"name":"tokenholder","image":"nginx:1.27-alpine","resources":{"requests":{"cpu":"25m","memory":"32Mi"},"limits":{"cpu":"100m","memory":"64Mi"}}}]}}'
kubectl -n team-a wait --for=condition=Ready pod/tokenholder --timeout=120s
kubectl -n team-a exec tokenholder -- cat /var/run/secrets/kubernetes.io/serviceaccount/token | cut -d. -f2 | tr '_-' '/+' | base64 -d 2>/dev/null | jq '{aud, exp, "kubernetes.io": ."kubernetes.io"}'
kubectl -n team-a delete pod tokenholder
kubectl -n team-a delete serviceaccount reporter

outputcaptured 2026-09-12

$ kubectl -n team-a create serviceaccount reporter --dry-run=client -o yaml | kubectl apply -f -
serviceaccount/reporter created
$ TOKEN=$(kubectl -n team-a create token reporter --duration=10m)
$ echo "$TOKEN" | cut -d. -f2 | tr '_-' '/+' | base64 -d 2>/dev/null | jq
{
  "aud": [
    "https://kubernetes.default.svc.cluster.local"
  ],
  "exp": 1789239113,
  "iat": 1789238513,
  "iss": "https://kubernetes.default.svc.cluster.local",
  "jti": "4a9851ec-3d17-4c59-aaa3-c5395ace326f",
  "kubernetes.io": {
    "namespace": "team-a",
    "serviceaccount": {
      "name": "reporter",
      "uid": "dcf3e81b-8822-4a69-81a1-dd5be70d6c44"
    }
  },
  "nbf": 1789238513,
  "sub": "system:serviceaccount:team-a:reporter"
}
$ kubectl -n team-a run tokenholder --image=nginx:1.27-alpine --restart=Never --overrides='{"spec":{"serviceAccountName":"reporter","containers":[{"name":"tokenholder","image":"nginx:1.27-alpine","resources":{"requests":{"cpu":"25m","memory":"32Mi"},"limits":{"cpu":"100m","memory":"64Mi"}}}]}}'
Warning: would violate PodSecurity "restricted:latest": allowPrivilegeEscalation != false (container "tokenholder" must set securityContext.allowPrivilegeEscalation=false), unrestricted capabilities (container "tokenholder" must set securityContext.capabilities.drop=["ALL"]), runAsNonRoot != true (pod or container "tokenholder" must set securityContext.runAsNonRoot=true), seccompProfile (pod or container "tokenholder" must set securityContext.seccompProfile.type to "RuntimeDefault" or "Localhost")
pod/tokenholder created
$ kubectl -n team-a wait --for=condition=Ready pod/tokenholder --timeout=120s
pod/tokenholder condition met
$ kubectl -n team-a exec tokenholder -- cat /var/run/secrets/kubernetes.io/serviceaccount/token | cut -d. -f2 | tr '_-' '/+' | base64 -d 2>/dev/null | jq '{aud, exp, "kubernetes.io": ."kubernetes.io"}'
{
  "aud": [
    "https://kubernetes.default.svc.cluster.local"
  ],
  "exp": 1820774514,
  "kubernetes.io": {
    "namespace": "team-a",
    "node": {
      "name": "cnpe-worker2",
      "uid": "b6be3349-5691-412e-9523-7ba669580823"
    },
    "pod": {
      "name": "tokenholder",
      "uid": "447d638e-ffcf-41da-a1d7-5bc0b7a33a23"
    },
    "serviceaccount": {
      "name": "reporter",
      "uid": "dcf3e81b-8822-4a69-81a1-dd5be70d6c44"
    },
    "warnafter": 1789242121
  }
}
$ kubectl -n team-a delete pod tokenholder
pod "tokenholder" deleted from team-a namespace
$ kubectl -n team-a delete serviceaccount reporter
serviceaccount "reporter" deleted from team-a namespace

verify: both tokens carry an exp and an aud, and the one projected into the pod additionally names the pod under kubernetes.io.

ESO's refresh policy decides whether the Kubernetes Secret tracks the external store or is a one-time copy. Combined with an immutable target, "CreatedOnce" is how you stop a rotation from breaking a running workload without a deploy.

# start from nothing: an immutable target from a previous run cannot be updated
kubectl delete externalsecret pg-once pg-periodic --ignore-not-found
kubectl delete secret pg-once pg-periodic --ignore-not-found
kubectl delete secretstore fake-store --ignore-not-found
kubectl apply -f - <<'EOF'
apiVersion: external-secrets.io/v1
kind: SecretStore
metadata: { name: fake-store, namespace: default }
spec:
  provider:
    fake:
      data:
        - key: pg/password
          value: first-value
---
apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata: { name: pg-once, namespace: default }
spec:
  refreshPolicy: CreatedOnce
  secretStoreRef: { name: fake-store, kind: SecretStore }
  target:
    name: pg-once
    immutable: true
  data:
    - secretKey: password
      remoteRef: { key: pg/password }
EOF
sleep 20
kubectl get secret pg-once -o jsonpath='{.data.password}' | base64 -d; echo
kubectl patch secretstore fake-store --type merge -p '{"spec":{"provider":{"fake":{"data":[{"key":"pg/password","value":"second-value"}]}}}}'
sleep 60
kubectl get secret pg-once -o jsonpath='{.data.password}' | base64 -d; echo
kubectl apply -f - <<'EOF'
apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata: { name: pg-periodic, namespace: default }
spec:
  refreshPolicy: Periodic
  refreshInterval: 30s
  secretStoreRef: { name: fake-store, kind: SecretStore }
  target: { name: pg-periodic }
  data:
    - secretKey: password
      remoteRef: { key: pg/password }
EOF
sleep 20
kubectl get secret pg-periodic -o jsonpath='{.data.password}' | base64 -d; echo
kubectl patch secretstore fake-store --type merge -p '{"spec":{"provider":{"fake":{"data":[{"key":"pg/password","value":"third-value"}]}}}}'
sleep 60
kubectl get secret pg-periodic -o jsonpath='{.data.password}' | base64 -d; echo
kubectl delete externalsecret pg-once pg-periodic
kubectl delete secret pg-once pg-periodic --ignore-not-found
kubectl delete secretstore fake-store --ignore-not-found

outputcaptured 2026-09-12

$ # start from nothing: an immutable target from a previous run cannot be updated
$ kubectl delete externalsecret pg-once pg-periodic --ignore-not-found
$ kubectl delete secret pg-once pg-periodic --ignore-not-found
$ kubectl delete secretstore fake-store --ignore-not-found
secretstore.external-secrets.io "fake-store" deleted from default namespace
$ kubectl apply -f - <<'EOF'
apiVersion: external-secrets.io/v1
kind: SecretStore
metadata: { name: fake-store, namespace: default }
spec:
  provider:
    fake:
      data:
        - key: pg/password
          value: first-value
---
apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata: { name: pg-once, namespace: default }
spec:
  refreshPolicy: CreatedOnce
  secretStoreRef: { name: fake-store, kind: SecretStore }
  target:
    name: pg-once
    immutable: true
  data:
    - secretKey: password
      remoteRef: { key: pg/password }
EOF
Warning: store fake-store isn't currently maintained. Please plan and prepare accordingly.
secretstore.external-secrets.io/fake-store created
externalsecret.external-secrets.io/pg-once created
$ sleep 20
$ kubectl get secret pg-once -o jsonpath='{.data.password}' | base64 -d; echo
first-value
$ kubectl patch secretstore fake-store --type merge -p '{"spec":{"provider":{"fake":{"data":[{"key":"pg/password","value":"second-value"}]}}}}'
Warning: store fake-store isn't currently maintained. Please plan and prepare accordingly.
secretstore.external-secrets.io/fake-store patched
$ sleep 60
$ kubectl get secret pg-once -o jsonpath='{.data.password}' | base64 -d; echo
first-value
$ kubectl apply -f - <<'EOF'
apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata: { name: pg-periodic, namespace: default }
spec:
  refreshPolicy: Periodic
  refreshInterval: 30s
  secretStoreRef: { name: fake-store, kind: SecretStore }
  target: { name: pg-periodic }
  data:
    - secretKey: password
      remoteRef: { key: pg/password }
EOF
externalsecret.external-secrets.io/pg-periodic created
$ sleep 20
$ kubectl get secret pg-periodic -o jsonpath='{.data.password}' | base64 -d; echo
second-value
$ kubectl patch secretstore fake-store --type merge -p '{"spec":{"provider":{"fake":{"data":[{"key":"pg/password","value":"third-value"}]}}}}'
Warning: store fake-store isn't currently maintained. Please plan and prepare accordingly.
secretstore.external-secrets.io/fake-store patched
$ sleep 60
$ kubectl get secret pg-periodic -o jsonpath='{.data.password}' | base64 -d; echo
third-value
$ kubectl delete externalsecret pg-once pg-periodic
externalsecret.external-secrets.io "pg-once" deleted from default namespace
externalsecret.external-secrets.io "pg-periodic" deleted from default namespace
$ kubectl delete secret pg-once pg-periodic --ignore-not-found
$ kubectl delete secretstore fake-store --ignore-not-found
secretstore.external-secrets.io "fake-store" deleted from default namespace

verify: the CreatedOnce secret keeps its first value through two rotations and the Periodic one follows within an interval.

Real secret stores hold JSON blobs, not tidy key-value pairs. dataFrom.extract unpacks one and a target template assembles exactly the string the application expects, so the app needs no knowledge of the store.

# start from nothing: an immutable target from a previous run cannot be updated
kubectl delete externalsecret pg-dsn --ignore-not-found
kubectl delete secret pg-dsn --ignore-not-found
kubectl delete secretstore json-store --ignore-not-found
kubectl apply -f - <<'EOF'
apiVersion: external-secrets.io/v1
kind: SecretStore
metadata: { name: json-store, namespace: default }
spec:
  provider:
    fake:
      data:
        - key: pg/creds
          value: '{"username":"app","password":"s3cr3t","host":"pg-rw.team-a.svc","port":"5432","db":"orders"}'
---
apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata: { name: pg-dsn, namespace: default }
spec:
  refreshInterval: 1m
  secretStoreRef: { name: json-store, kind: SecretStore }
  target:
    name: pg-dsn
    template:
      engineVersion: v2
      data:
        DATABASE_URL: "postgres://{{ .username }}:{{ .password }}@{{ .host }}:{{ .port }}/{{ .db }}?sslmode=require"
  dataFrom:
    - extract: { key: pg/creds }
EOF
sleep 20
kubectl get externalsecret pg-dsn -o jsonpath='{.status.conditions}' | jq '.[] | {type, status, reason}'
kubectl get secret pg-dsn -o jsonpath='{.data.DATABASE_URL}' | base64 -d; echo
kubectl delete externalsecret pg-dsn
kubectl delete secretstore json-store

outputcaptured 2026-09-13

$ # start from nothing: an immutable target from a previous run cannot be updated
$ kubectl delete externalsecret pg-dsn --ignore-not-found
$ kubectl delete secret pg-dsn --ignore-not-found
$ kubectl delete secretstore json-store --ignore-not-found
$ kubectl apply -f - <<'EOF'
apiVersion: external-secrets.io/v1
kind: SecretStore
metadata: { name: json-store, namespace: default }
spec:
  provider:
    fake:
      data:
        - key: pg/creds
          value: '{"username":"app","password":"s3cr3t","host":"pg-rw.team-a.svc","port":"5432","db":"orders"}'
---
apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata: { name: pg-dsn, namespace: default }
spec:
  refreshInterval: 1m
  secretStoreRef: { name: json-store, kind: SecretStore }
  target:
    name: pg-dsn
    template:
      engineVersion: v2
      data:
        DATABASE_URL: "postgres://{{ .username }}:{{ .password }}@{{ .host }}:{{ .port }}/{{ .db }}?sslmode=require"
  dataFrom:
    - extract: { key: pg/creds }
EOF
Warning: store json-store isn't currently maintained. Please plan and prepare accordingly.
secretstore.external-secrets.io/json-store created
externalsecret.external-secrets.io/pg-dsn created
$ sleep 20
$ kubectl get externalsecret pg-dsn -o jsonpath='{.status.conditions}' | jq '.[] | {type, status, reason}'
{
  "type": "Ready",
  "status": "True",
  "reason": "SecretSynced"
}
$ kubectl get secret pg-dsn -o jsonpath='{.data.DATABASE_URL}' | base64 -d; echo
postgres://app:s3cr3t@pg-rw.team-a.svc:5432/orders?sslmode=require
$ kubectl delete externalsecret pg-dsn
externalsecret.external-secrets.io "pg-dsn" deleted from default namespace
$ kubectl delete secretstore json-store
secretstore.external-secrets.io "json-store" deleted from default namespace

verify: the materialized Secret holds one key containing a complete DSN assembled from five fields you never wrote into the cluster.

A SealedSecret is bound to its name and namespace by default, which is exactly right and occasionally inconvenient. The scope annotation trades some of that binding for portability, and the failure when you get it wrong is in the controller's log.

echo -n 'strict-value' | kubeseal --raw --namespace default --name strict-secret --controller-namespace kube-system --controller-name sealed-secrets > /tmp/strict.b64
cat > /tmp/strict.yaml <<EOF
apiVersion: bitnami.com/v1alpha1
kind: SealedSecret
metadata: { name: strict-secret, namespace: default }
spec:
  encryptedData:
    value: $(cat /tmp/strict.b64)
EOF
kubectl apply -f /tmp/strict.yaml
sleep 10
kubectl get secret strict-secret -o jsonpath='{.data.value}' | base64 -d; echo
sed 's/name: strict-secret/name: renamed-secret/' /tmp/strict.yaml | kubectl apply -f -
sleep 10
kubectl get secret renamed-secret 2>/dev/null || echo 'not decrypted'
kubectl -n kube-system logs deploy/sealed-secrets --tail=10
echo -n 'wide-value' | kubeseal --raw --scope namespace-wide --namespace default --controller-namespace kube-system --controller-name sealed-secrets > /tmp/wide.b64
cat > /tmp/wide.yaml <<EOF
apiVersion: bitnami.com/v1alpha1
kind: SealedSecret
metadata:
  name: wide-secret
  namespace: default
  annotations: { sealedsecrets.bitnami.com/namespace-wide: "true" }
spec:
  encryptedData:
    value: $(cat /tmp/wide.b64)
EOF
kubectl apply -f /tmp/wide.yaml
sleep 10
sed 's/name: wide-secret/name: wide-renamed/' /tmp/wide.yaml | kubectl apply -f -
sleep 10
kubectl get secret wide-renamed -o jsonpath='{.data.value}' | base64 -d; echo
kubectl delete sealedsecret strict-secret renamed-secret wide-secret wide-renamed --ignore-not-found

outputcaptured 2026-09-12

$ echo -n 'strict-value' | kubeseal --raw --namespace default --name strict-secret --controller-namespace kube-system --controller-name sealed-secrets > /tmp/strict.b64
$ cat > /tmp/strict.yaml <<EOF
apiVersion: bitnami.com/v1alpha1
kind: SealedSecret
metadata: { name: strict-secret, namespace: default }
spec:
  encryptedData:
    value: $(cat /tmp/strict.b64)
EOF
$ kubectl apply -f /tmp/strict.yaml
sealedsecret.bitnami.com/strict-secret created
$ sleep 10
$ kubectl get secret strict-secret -o jsonpath='{.data.value}' | base64 -d; echo
strict-value
$ sed 's/name: strict-secret/name: renamed-secret/' /tmp/strict.yaml | kubectl apply -f -
sealedsecret.bitnami.com/renamed-secret created
$ sleep 10
$ kubectl get secret renamed-secret 2>/dev/null || echo 'not decrypted'
not decrypted
$ kubectl -n kube-system logs deploy/sealed-secrets --tail=10
time=2026-09-12T19:44:52.637Z level=INFO msg=Updating key=default/renamed-secret
time=2026-09-12T19:44:52.644Z level=ERROR msg="Error updating, will retry" key=default/renamed-secret error="no key could decrypt secret (value)"
time=2026-09-12T19:44:52.644Z level=INFO msg="Event(v1.ObjectReference{Kind:\"SealedSecret\", Namespace:\"default\", Name:\"renamed-secret\", UID:\"9f10a4d2-f72f-4f66-ad14-12352277718c\", APIVersion:\"bitnami.com/v1alpha1\", ResourceVersion:\"27307\", FieldPath:\"\"}): type: 'Warning' reason: 'ErrUnsealFailed' Failed to unseal: no key could decrypt secret (value)"
time=2026-09-12T19:44:52.685Z level=INFO msg=Updating key=default/renamed-secret
time=2026-09-12T19:44:52.692Z level=ERROR msg="Error updating, will retry" key=default/renamed-secret error="no key could decrypt secret (value)"
time=2026-09-12T19:44:52.692Z level=INFO msg="Event(v1.ObjectReference{Kind:\"SealedSecret\", Namespace:\"default\", Name:\"renamed-secret\", UID:\"9f10a4d2-f72f-4f66-ad14-12352277718c\", APIVersion:\"bitnami.com/v1alpha1\", ResourceVersion:\"27307\", FieldPath:\"\"}): type: 'Warning' reason: 'ErrUnsealFailed' Failed to unseal: no key could decrypt secret (value)"
time=2026-09-12T19:44:52.773Z level=INFO msg=Updating key=default/renamed-secret
time=2026-09-12T19:44:52.780Z level=ERROR msg="Error updating, giving up" key=default/renamed-secret error="no key could decrypt secret (value)"
E0912 19:44:52.780710       1 controller.go:320] "Unhandled Error" err="no key could decrypt secret (value)" logger="UnhandledError"
time=2026-09-12T19:44:52.780Z level=INFO msg="Event(v1.ObjectReference{Kind:\"SealedSecret\", Namespace:\"default\", Name:\"renamed-secret\", UID:\"9f10a4d2-f72f-4f66-ad14-12352277718c\", APIVersion:\"bitnami.com/v1alpha1\", ResourceVersion:\"27307\", FieldPath:\"\"}): type: 'Warning' reason: 'ErrUnsealFailed' Failed to unseal: no key could decrypt secret (value)"
$ echo -n 'wide-value' | kubeseal --raw --scope namespace-wide --namespace default --controller-namespace kube-system --controller-name sealed-secrets > /tmp/wide.b64
$ cat > /tmp/wide.yaml <<EOF
apiVersion: bitnami.com/v1alpha1
kind: SealedSecret
metadata:
  name: wide-secret
  namespace: default
  annotations: { sealedsecrets.bitnami.com/namespace-wide: "true" }
spec:
  encryptedData:
    value: $(cat /tmp/wide.b64)
EOF
$ kubectl apply -f /tmp/wide.yaml
sealedsecret.bitnami.com/wide-secret created
$ sleep 10
$ sed 's/name: wide-secret/name: wide-renamed/' /tmp/wide.yaml | kubectl apply -f -
sealedsecret.bitnami.com/wide-renamed created
$ sleep 10
$ kubectl get secret wide-renamed -o jsonpath='{.data.value}' | base64 -d; echo
wide-value
$ kubectl delete sealedsecret strict-secret renamed-secret wide-secret wide-renamed --ignore-not-found
sealedsecret.bitnami.com "strict-secret" deleted from default namespace
sealedsecret.bitnami.com "renamed-secret" deleted from default namespace
sealedsecret.bitnami.com "wide-secret" deleted from default namespace
sealedsecret.bitnami.com "wide-renamed" deleted from default namespace

verify: the strict secret refuses to decrypt under a new name with a decryption error in the controller log, and the namespace-wide one decrypts under any name in that namespace. Say what cluster-wide would give up, and why you would rarely want it.

SOPS is the other answer to secrets in git: the file stays readable except for the values, and Flux decrypts it at apply time with a key that only the cluster has. The whole loop is five commands.

START=$PWD   # the cd below would otherwise follow you into every later command
mkdir -p "$HOME/.local/bin"; export PATH="$HOME/.local/bin:$PATH"
curl -sL "$(curl -s https://api.github.com/repos/FiloSottile/age/releases/latest | jq -r '.assets[] | select(.name | endswith("linux-amd64.tar.gz")) | .browser_download_url')" | tar xz -C /tmp && install /tmp/age/age /tmp/age/age-keygen "$HOME/.local/bin/"
curl -sLo "$HOME/.local/bin/sops" "$(curl -s https://api.github.com/repos/getsops/sops/releases/latest | jq -r '.assets[] | select(.name | endswith("linux.amd64")) | .browser_download_url')" && chmod +x "$HOME/.local/bin/sops"
age-keygen --version; sops --version | head -1
rm -f /tmp/sops.agekey && age-keygen -o /tmp/sops.agekey
PUB=$(grep 'public key' /tmp/sops.agekey | awk '{print $NF}'); echo "recipient: $PUB"
kubectl -n flux-system create secret generic sops-age --from-file=age.agekey=/tmp/sops.agekey --dry-run=client -o yaml | kubectl apply -f -
rm -rf /tmp/platform-sops && git clone "http://lab:${GITEA_PASS}@gitea.lab:3000/lab/platform.git" /tmp/platform-sops && cd /tmp/platform-sops
mkdir -p secrets
cat > .sops.yaml <<EOF
creation_rules:
  - path_regex: secrets/.*\.yaml$
    encrypted_regex: '^(data|stringData)$'
    age: $PUB
EOF
kubectl -n flux-demo create secret generic app-config --from-literal=api-key=super-secret --dry-run=client -o yaml > secrets/app-config.yaml
sops --encrypt --in-place secrets/app-config.yaml
head -12 secrets/app-config.yaml
git add .sops.yaml secrets && git commit -m 'sops-encrypted secret' && git push
kubectl create ns flux-demo --dry-run=client -o yaml | kubectl apply -f -
kubectl apply -f - <<'EOF'
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata: { name: secrets, namespace: flux-system }
spec:
  interval: 1m
  prune: true
  sourceRef: { kind: GitRepository, name: platform }
  path: ./secrets
  decryption:
    provider: sops
    secretRef: { name: sops-age }
EOF
sleep 60
kubectl -n flux-demo get secret app-config -o jsonpath='{.data.api-key}' | base64 -d; echo
# put the lab back: this repo is shared with every other GitOps exercise on the site
flux delete kustomization secrets -s
sleep 10
kubectl -n flux-demo get secret app-config --ignore-not-found
git rm -r --quiet secrets .sops.yaml && git commit -m 'remove the sops demo' && git push
kubectl -n flux-system delete secret sops-age
cd "$START"

outputcaptured 2026-09-13

$ START=$PWD   # the cd below would otherwise follow you into every later command
$ mkdir -p "$HOME/.local/bin"; export PATH="$HOME/.local/bin:$PATH"
$ curl -sL "$(curl -s https://api.github.com/repos/FiloSottile/age/releases/latest | jq -r '.assets[] | select(.name | endswith("linux-amd64.tar.gz")) | .browser_download_url')" | tar xz -C /tmp && install /tmp/age/age /tmp/age/age-keygen "$HOME/.local/bin/"
$ curl -sLo "$HOME/.local/bin/sops" "$(curl -s https://api.github.com/repos/getsops/sops/releases/latest | jq -r '.assets[] | select(.name | endswith("linux.amd64")) | .browser_download_url')" && chmod +x "$HOME/.local/bin/sops"
$ age-keygen --version; sops --version | head -1
v1.3.2
sops 3.13.3 (latest)
$ rm -f /tmp/sops.agekey && age-keygen -o /tmp/sops.agekey
Public key: age13mwr72a08y8qhvusarepklye8h8j9ffw3pxuppgutdqmkj9c3drqqlumxf
$ PUB=$(grep 'public key' /tmp/sops.agekey | awk '{print $NF}'); echo "recipient: $PUB"
recipient: age13mwr72a08y8qhvusarepklye8h8j9ffw3pxuppgutdqmkj9c3drqqlumxf
$ kubectl -n flux-system create secret generic sops-age --from-file=age.agekey=/tmp/sops.agekey --dry-run=client -o yaml | kubectl apply -f -
secret/sops-age created
$ rm -rf /tmp/platform-sops && git clone "http://lab:${GITEA_PASS}@gitea.lab:3000/lab/platform.git" /tmp/platform-sops && cd /tmp/platform-sops
Cloning into '/tmp/platform-sops'...
$ mkdir -p secrets
$ cat > .sops.yaml <<EOF
creation_rules:
  - path_regex: secrets/.*\.yaml$
    encrypted_regex: '^(data|stringData)$'
    age: $PUB
EOF
$ kubectl -n flux-demo create secret generic app-config --from-literal=api-key=super-secret --dry-run=client -o yaml > secrets/app-config.yaml
$ sops --encrypt --in-place secrets/app-config.yaml
$ head -12 secrets/app-config.yaml
apiVersion: v1
data:
    api-key: ENC[AES256_GCM,data:Quby+vT4+2SjztVEfihNsg==,iv:toiP/Jmsl8B7MqbLg9I4gXytnuQX6RS9vSy4T1gb/Zw=,tag:TlhhcNVt0Gt5S7pkgRHQYQ==,type:str]
kind: Secret
metadata:
    name: app-config
    namespace: flux-demo
sops:
    age:
        - enc: |
            -----BEGIN AGE ENCRYPTED FILE-----
            YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSAvTmFIZE1mWjgxUGNoTUFp
$ git add .sops.yaml secrets && git commit -m 'sops-encrypted secret' && git push
[main fa86adb] sops-encrypted secret
 2 files changed, 26 insertions(+)
 create mode 100644 .sops.yaml
 create mode 100644 secrets/app-config.yaml
To http://gitea.lab:3000/lab/platform.git
   a816a73..fa86adb  main -> main
$ kubectl create ns flux-demo --dry-run=client -o yaml | kubectl apply -f -
namespace/flux-demo created
$ kubectl apply -f - <<'EOF'
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata: { name: secrets, namespace: flux-system }
spec:
  interval: 1m
  prune: true
  sourceRef: { kind: GitRepository, name: platform }
  path: ./secrets
  decryption:
    provider: sops
    secretRef: { name: sops-age }
EOF
kustomization.kustomize.toolkit.fluxcd.io/secrets created
$ sleep 60
$ kubectl -n flux-demo get secret app-config -o jsonpath='{.data.api-key}' | base64 -d; echo
super-secret
$ # put the lab back: this repo is shared with every other GitOps exercise on the site
$ flux delete kustomization secrets -s
► deleting kustomization secrets in flux-system namespace
✔ kustomization deleted
$ sleep 10
$ kubectl -n flux-demo get secret app-config --ignore-not-found
$ git rm -r --quiet secrets .sops.yaml && git commit -m 'remove the sops demo' && git push
[main 4a5f1ae] remove the sops demo
 2 files changed, 26 deletions(-)
 delete mode 100644 .sops.yaml
 delete mode 100644 secrets/app-config.yaml
To http://gitea.lab:3000/lab/platform.git
   fa86adb..4a5f1ae  main -> main
$ kubectl -n flux-system delete secret sops-age
secret "sops-age" deleted from flux-system namespace
$ cd "$START"

verify: the committed file shows the key names in the clear and the values as ciphertext, and the Secret in the cluster holds the plaintext. Then the teardown: deleting the Kustomization prunes the Secret it created, and the second commit takes the demo back out of the platform repo, which every other GitOps exercise on this site also reconciles from.

Self-check

answer before opening

Define "developer" once and grant it in twelve namespaces. What objects, and how many?

One ClusterRole plus twelve RoleBindings (each referencing that ClusterRole, each in its namespace). Not a ClusterRoleBinding: that would grant everywhere and fail the "and not in the other namespace" check.

A user can get a pod by name but kubectl get pods fails. Why?

list is a separate verb from get, and a rule with resourceNames cannot grant a plain list. Add list (and watch if anything streams), accepting that list exposes every object of that kind in the namespace.

How do you let a tenant read pod logs but never exec into a pod?

Grant get on pods/log and do not grant create on pods/exec. Subresources are distinct resource strings, which is what makes this expressible at all.

You ship a new CRD. How do existing view users get read access without editing the built-in role?

Create a ClusterRole with read verbs on your kind and label it rbac.authorization.k8s.io/aggregate-to-view: "true". Aggregation folds it into the built-in role automatically, the standard platform-team move when adding an API.

Sealed Secrets versus ESO: which would you pick for a regulated environment with an existing vault, and why?

ESO: the vault stays the single source of truth with its own audit and rotation, and git holds only references; nothing secret is ever committed, encrypted or not. Sealed Secrets suits clusters with no external store, at the cost of managing (and backing up) the sealing key yourself.

Is a Kubernetes Secret encrypted?

Not by itself: base64 is encoding. At rest it is only encrypted if the API server is configured with an EncryptionConfiguration (optionally KMS-backed); in transit it is protected by TLS. Access control (who has get secrets) is doing most of the actual work.

A namespace admin with the built-in admin role cannot create a RoleBinding to a custom ClusterRole that grants nodes read. Why, and what is the least-privilege fix?

Escalation prevention: you can only bind permissions you hold, and admin does not include cluster-scoped nodes. Grant the bind verb on clusterroles with resourceNames naming that one ClusterRole. Granting the node permission itself or cluster-admin would also work and would be the wrong answer.

An OIDC user in group platform-team is bound via a ClusterRoleBinding to that group and still gets Forbidden. First command, likely cause?

kubectl auth whoami shows the groups as the API server sees them, prefixed by --oidc-groups-prefix (or the claimMappings.groups.prefix in the structured config), so the real group is something like oidc:platform-team. Bind to that string. The same check catches a username prefix and a wrong groups claim name.

You rotate the aescbc key in EncryptionConfiguration. Order of operations, and what happens if you simply replace the key?

Add the new key as the first provider while keeping the old key second, restart the API servers, rewrite every Secret so it is re-encrypted (kubectl get secrets -A -o json | kubectl replace -f -), then drop the old key. Replacing the key outright leaves every existing Secret unreadable, which surfaces as API server errors on get secrets, not at startup.

ESO: a database password rotates in Vault every hour, the ExternalSecret has refreshInterval: 0, and pods keep failing to log in. Two fixes and their trade-offs?

refreshInterval: 0 means create once and never update, so the Secret holds the first value forever. Set a refreshInterval shorter than the rotation period (Periodic policy), and make the workload reload (a restart trigger such as Reloader, or read the file mount rather than env vars). Alternatively, if the value must not change under a running pod, keep CreatedOnce plus immutable: true and roll the pods deliberately.

Docs to know your way around

study time, not exam time

kubernetes.io: Using RBAC Authorization (the RoleBinding-to-ClusterRole pattern and aggregation are both spelled out there); Managing Service Accounts; Encrypting Secret Data at Rest.
sealed-secrets and external-secrets docs: one page each on their CRDs; the fake provider is under ESO's provider list.
Offline: kubectl auth can-i --list, kubectl api-resources --verbs=list, kubectl create role --help (its examples are a rules cheat sheet).
kubernetes.io/docs/reference/access-authn-authz/rbac, sections "Privilege escalation prevention and bootstrapping" and "Default roles and role bindings": the two parts of the page a Forbidden puzzle sends you to.
kubernetes.io/docs/tasks/administer-cluster/encrypt-data (providers table, rotation steps) and /docs/reference/access-authn-authz/authentication (OIDC flags, AuthenticationConfiguration, SelfSubjectReview).
external-secrets.io/latest/api/externalsecret (refresh and creation policies) and /guides/templating; fluxcd.io/flux/guides/mozilla-sops for the Kustomization decryption block.

free before 5.2make down-gitops