Mock exam
Fifteen tasks, 120 minutes, 100 points, weighted like the real blueprint (15/25/25/20/15). Layers required: make full (everything on the main cluster; no mesh, no portal). Also have the 2.2 ApplicationSet applied (kubectl apply -f examples/argocd-appset.yaml) and the demo-app Dockerfile from section 2.4 pushed; both are part of the curriculum's normal flow.
The clock
The rules: start the timer, no peeking at curriculum sections or the answers below, and grade only at the end with the commands in the grading section. Docs and --help are allowed, as on the real exam. Anything a grading command cannot see does not count; the real exam grades the same way. Target: 70+ and no domain at zero. If a task stalls, note the time, move on, come back.
Tasks
Create namespace team-x labeled pod-security.kubernetes.io/enforce=baseline, with a ResourceQuota mock-quota (requests.cpu 1, requests.memory 2Gi, pods 5) and a LimitRange mock-limits defaulting containers to 100m/128Mi requests and 200m/256Mi limits.
In team-x, create deployment web (image ghcr.io/nginxinc/nginx-unprivileged:1.27-alpine, port 8080), 4 replicas, each container requesting exactly 25m CPU and 32Mi memory, spread across zones with a topologySpreadConstraint (maxSkew: 1, whenUnsatisfiable: DoNotSchedule on topology.kubernetes.io/zone). All 4 must become Ready.
Through git only (no kubectl mutations): set the staging overlay of demo-app to 2 replicas and get it live via Argo CD. The live deployment is staging-demo in team-a.
Create a Flux Kustomization mock-demo on the existing GitRepository/platform deploying path ./demo-app/base into namespace mock-flux (create it), prune enabled, interval 1m. Both replicas Ready.
Apply examples/rollouts/canary.yaml, wait for the rollout demo in team-a to be Healthy, trigger a canary with image tag 1.26-alpine, then abort it while it is mid-steps (the metric analysis may abort it for you first, since the demo app exposes no http_requests_total; run the abort yourself anyway). The rollout must end aborted with the stable version at 1.27-alpine.
Create a namespaced CRD mockapps.mock.lab (kind MockApp, plural mockapps) whose spec requires field tier (string, enum: dev, prod) and shows it in a printer column named TIER. Create instance one (tier: dev) in default. A MockApp with tier: staging must be rejected by the API server.
Using the lab's existing AppEnvironment platform API, provision an environment team-y-dev in default for team team-y with cpuQuota "3" and memoryQuota 6Gi, and wait until it is fully reconciled.
Create and run an Argo Workflow in default (any name, generateName fine) that creates ConfigMap mock-result in default with data result=done, running under a ServiceAccount that you grant exactly the rights this needs. The workflow must reach Succeeded.
Deploy quay.io/brancz/prometheus-example-app:v0.5.0 in default as deployment+service mock-metrics (port 8080) and get Prometheus scraping it via a ServiceMonitor. The target must reach state up.
Create a PrometheusRule that defines alert MockDown, firing (after 1m) whenever mock-metrics has zero healthy scrape targets. Prove it can fire by scaling mock-metrics to 0 and letting it reach Alertmanager.
Run make break. Diagnose it, write the root cause in one line to /tmp/mock-diagnosis.txt before revealing anything, then fix the cluster yourself (no make break-fix). Score it only if make break-answer matches your written diagnosis and the fault's symptom is demonstrably gone.
Create ServiceAccount auditor in team-b that can get and list pods and deployments in team-b and nothing else (no create, no other namespaces).
Create namespace policy-test, then make the cluster reject any new pod there whose containers lack CPU or memory requests, using the policy engine already installed. (Flip it back after grading. If your first instinct was to demonstrate this in team-b, work out why that can never show a rejection before reading the answer.)
From the API audit log, extract every successful delete performed today by the admin user into /tmp/mock-audit.json, one JSON event per line, each containing verb, user.username, and objectRef.
Generate a cosign key pair at /tmp/mock-cosign.key / /tmp/mock-cosign.pub (grading reads the public key from that path), sign the image localhost:5001/demo:v1 by digest, and verify the signature. Both sign and verify must succeed against the registry.
Grading
Run all of it; each line prints what earns the points. From the repo, make grade runs this block verbatim: it extracts it from this very page, so paper and grader cannot drift.
# 1
kubectl get ns team-x -o jsonpath='{.metadata.labels.pod-security\.kubernetes\.io/enforce}{"\n"}' # baseline
kubectl -n team-x get resourcequota mock-quota -o jsonpath='{.spec.hard}{"\n"}' # cpu 1, mem 2Gi, pods 5
kubectl -n team-x run graded --image=busybox:1.37 --restart=Never --dry-run=client -o yaml -- sleep 30 | kubectl apply -f - ; sleep 5
kubectl -n team-x get pod graded -o jsonpath='{.spec.containers[0].resources}{"\n"}' # 100m/128Mi defaults injected
# 2
kubectl -n team-x get deploy web -o jsonpath='{.status.readyReplicas}{"\n"}' # 4
kubectl -n team-x get pods -l app=web -o custom-columns=NODE:.spec.nodeName --no-headers | sort | uniq -c # 2+2 across zone nodes
# 3
kubectl -n team-a get deploy staging-demo -o jsonpath='{.spec.replicas}{"\n"}' # 2
kubectl -n argocd get app demo-staging -o jsonpath='{.status.sync.status}{"\n"}' # Synced (git did it, not kubectl)
# 4
flux get kustomization mock-demo # Ready True
kubectl -n mock-flux get deploy demo -o jsonpath='{.status.readyReplicas}{"\n"}' # 2
# 5
kubectl -n team-a get rollout demo -o jsonpath='{.status.abort}{"\n"}' # true
kubectl -n team-a get rollout demo -o jsonpath='{.spec.template.spec.containers[0].image}{"\n"}' # ...1.26-alpine requested
kubectl argo rollouts status demo -n team-a --timeout 10s 2>&1 | head -1 # Degraded/aborted, stable serving old
# 6
kubectl get crd mockapps.mock.lab -o jsonpath='{.status.conditions[?(@.type=="Established")].status}{"\n"}' # True
kubectl get mockapp one # TIER column shows dev
kubectl apply -f <(kubectl get mockapp one -o json | jq '.metadata.name="bad" | .spec.tier="staging"') 2>&1 | grep -i unsupported # rejection
# 7
kubectl get appenvironment team-y-dev -o jsonpath='{.status.conditions[?(@.type=="Ready")].status}{"\n"}' # True
kubectl -n team-y get resourcequota tenant-quota -o jsonpath='{.spec.hard}{"\n"}' # cpu 3, memory 6Gi
# 8
kubectl get workflow -o jsonpath='{.items[?(@.status.phase=="Succeeded")].metadata.name}{"\n"}' # your workflow
kubectl get cm mock-result -o jsonpath='{.data.result}{"\n"}' # done
# 9
kubectl -n monitoring port-forward svc/prometheus-kube-prometheus-prometheus 19090:9090 & sleep 3
curl -sG 'http://localhost:19090/api/v1/query' --data-urlencode 'query=max_over_time(up{job="mock-metrics"}[2h])' | jq -r '.data.result[0].value[1]' # 1 (holds even after task 10 scaled it to 0)
# 10 (after scaling to 0 and waiting out the for:)
kubectl -n monitoring port-forward svc/prometheus-kube-prometheus-alertmanager 19093:9093 & sleep 3
curl -s 'http://localhost:19093/api/v2/alerts?filter=alertname%3D%22MockDown%22' | jq -r '.[0].status.state' # active
# 11
cat /tmp/mock-diagnosis.txt && make break-answer # must agree; then show the symptom gone
# 12
kubectl auth can-i list pods --as=system:serviceaccount:team-b:auditor -n team-b # yes
kubectl auth can-i create pods --as=system:serviceaccount:team-b:auditor -n team-b # no
kubectl auth can-i list pods --as=system:serviceaccount:team-b:auditor -n team-a # no
# 13
kubectl -n policy-test run graded-naked --image=busybox:1.37 --restart=Never -- sleep 5 2>&1 | grep -ci 'request' # >=1 (rejected)
# 14
jq -e 'select(.verb=="delete") | .user.username, .objectRef.resource' /tmp/mock-audit.json | head # parses, right fields
# 15
cosign verify --key /tmp/mock-cosign.pub --allow-http-registry localhost:5001/demo@$(skopeo inspect --tls-verify=false docker://localhost:5001/demo:v1 | jq -r .Digest) 2>&1 | grep -c 'signatures were verified' # 1Answers
Key moves only; each maps to a curriculum section where the full treatment lives.
1 · namespace, quota, limit range
Three objects, section 1.4's pattern. The graded check is the LimitRange injecting defaults, so default: and defaultRequest: must both be set.
2 · spread deployment
Requests must be explicit or team-x's LimitRange (100m) would overshoot the quota at 4 replicas; 25m×4 fits under 1 CPU. Spread as in section 1.2. If pods pend, read the quota/scheduling event.
3 · change through git
Clone the platform repo with credentials (http://lab:$GITEA_PASS@gitea.lab:3000/lab/platform.git; anonymous push fails), set replicas in demo-app/overlays/staging/kustomization.yaml, push, let the ApplicationSet's automated sync land it, or argocd app sync demo-staging to skip the poll. Section 2.2.
4 · Flux Kustomization
flux create kustomization mock-demo --source=GitRepository/platform --path=./demo-app/base --target-namespace=mock-flux --prune=true --interval=1m after creating the namespace. Section 2.3.
5 · canary and abort
kubectl argo rollouts set image demo web=ghcr.io/nginxinc/nginx-unprivileged:1.26-alpine -n team-a, then kubectl argo rollouts abort demo -n team-a during the pause step. Aborted leaves spec at the new tag, stable ReplicaSet serving the old one, status.abort: true. Section 2.5.
6 · CRD with validation
Section 3.2's CRD skeleton with enum: [dev, prod] and an additionalPrinterColumns entry (name: TIER, jsonPath .spec.tier). The rejection sentence comes from the enum validator.
7 · AppEnvironment
Copy examples/crossplane/xr.yaml, change name/team/quotas, apply, kubectl get appenvironment -w until Ready. Section 3.5; crossplane resource trace if it sticks.
8 · workflow with RBAC
Section 3.4's pattern with a single resource template (action create, the ConfigMap manifest inline). The SA needs create on configmaps in default and create,patch on workflowtaskresults.argoproj.io (the executor's reporting channel; forget it and a correct workflow still fails), bound with Roles, named in spec.serviceAccountName.
9 · scrape a new target
Deployment + Service with a named port, ServiceMonitor selecting the Service's labels, endpoint port by name. This lab's Prometheus selects all monitors (selector {}); on a stock install you would also need the release label, so check the selector first. Section 4.1's second exercise walks through this task step by step. Grading queries max_over_time(up{job="mock-metrics"}[2h]) rather than live up because task 10 has you scale the target to zero: the point still counts while the target is down, as long as it scraped up within the two hours before you grade (so grade promptly; an early task 9 plus a long break can age the sample out of the window).
10 · alert that fires
expr: absent(up{job="mock-metrics"} == 1) (or count(up{...}==1) == 0 with or vector care), for: 1m. Scale to 0, wait ~2–3 min for evaluate + for: + group_wait. Section 4.2. (Rule selector logic is the same story as answer 9.)
11 · diagnose the fault
Section 4.6's method: blast radius, timeline, descend. The target is under 7 minutes.
12 · read-only service account
Role (get/list pods+deployments) + RoleBinding to the SA, both in team-b only. The third check fails if you reached for a ClusterRoleBinding. Section 5.1.
13 · enforce the policy
kubectl patch validatingpolicy require-resource-requests --type=merge -p '{"spec":{"validationActions":["Deny"]}}'; the policy ships in Audit. The task uses policy-test because the tenant namespaces have LimitRanges, and LimitRanger mutates requests in before validation ever runs, so team-b literally cannot produce a rejection. Section 5.2 makes a whole lesson of it. Flip back to ["Audit"].
14 · audit extraction
docker exec cnpe-control-plane sh -c 'cat /var/log/kubernetes/audit*.log' | jq -c --arg d "$(date -u +%F)" 'select(.verb=="delete" and .responseStatus.code<300 and (.user.username|test("admin")) and (.requestReceivedTimestamp|startswith($d)))' > /tmp/mock-audit.json. Section 5.4. Read all of today's files, not just audit.log: the apiserver rotates at 100MB, which a full lab fills in about an hour, so a delete from early in the exam has usually rotated out of the current file by grading time (the $d filter is what keeps "today" honest across those rotations).
15 · sign and verify
Section 5.6's exercise with the key at the graded path: cosign generate-key-pair --output-key-prefix /tmp/mock-cosign, sign by digest with --allow-http-registry -y, verify with the pub key. If demo:v1 doesn't exist, the 2.4 pipeline builds it (pushing as kind-registry:5000/demo:v1, which the host reads back as localhost:5001/demo:v1).
After scoring
Log the score per domain, not just the total; a domain at zero fails you in ways an average hides. Re-drill the weakest domain's sections, wait two days, then take mock exam 2 on a rebuilt lab (make down && make full gives you a clean grader): fifteen fresh tasks you have not half-memorized, scored separately. The second paper under time is the one that predicts the real thing.
Sweep the lab before any rebuild, and parents first: delete the Kustomization and the AppEnvironments, not the namespaces they manage, or their controllers rebuild what you just removed. The git revert comes first because it is the one line that still matters after make down && make full: task 3's replica change lives in the git server, which survives make down (only make nuke clears it), so left unreverted it pre-passes that task on every future paper.
# task 3 lives in git: put staging back to 1 replica or every future paper inherits the answer
source lab.env # for GITEA_PASS
tmp=$(mktemp -d) && git clone -q "http://lab:${GITEA_PASS}@gitea.lab:3000/lab/platform.git" "$tmp" && \
sed -i -E 's/(name: demo, count:) *[0-9]+/\1 1/' "$tmp/demo-app/overlays/staging/kustomization.yaml" && \
git -C "$tmp" commit -aqm "staging back to 1 replica" && git -C "$tmp" push -q; rm -rf "$tmp"
# controllers first, or they rebuild what you are deleting
flux delete kustomization mock-demo --silent # prune takes the mock-flux deployment with it
kubectl delete appenvironment team-y-dev --ignore-not-found # Crossplane deletes team-y for you
kubectl delete appenvironment drill-env --ignore-not-found # if task 11 dealt you the apis drill
kubectl -n team-a delete rollout demo --ignore-not-found
kubectl -n team-a delete analysistemplate success-rate --ignore-not-found
kubectl delete crd mockapps.mock.lab --ignore-not-found # instances go with it
# names below task text never fixed are yours; --ignore-not-found keeps the sweep going
kubectl delete wf --all # task 8's name was yours; this also clears 3.4's practice runs in default
kubectl delete sa mock-wf --ignore-not-found; kubectl delete role,rolebinding mock-wf --ignore-not-found
kubectl delete cm mock-result --ignore-not-found
kubectl delete deploy,svc,servicemonitor mock-metrics --ignore-not-found
# 4.2's version puts the rule in monitoring; kubectl get prometheusrule -A if you named it something else
kubectl delete prometheusrule mock-down --ignore-not-found
kubectl -n monitoring delete prometheusrule mock-down --ignore-not-found
kubectl -n team-b delete sa auditor --ignore-not-found; kubectl -n team-b delete role,rolebinding auditor --ignore-not-found
kubectl patch validatingpolicy require-resource-requests --type=merge -p '{"spec":{"validationActions":["Audit"]}}'
kubectl delete ns team-x mock-flux policy-test --ignore-not-found
# the signature on demo:v1 stays: the lab registry refuses manifest deletes (cosign clean
# reports UNSUPPORTED), and it is inert anyway: task 15 verifies against a fresh key
rm -f /tmp/mock-cosign.key /tmp/mock-cosign.pub /tmp/mock-diagnosis.txt /tmp/mock-audit.json
pkill -f 'port-forward.*1909[03]' # the two grading port-forwards, if you pasted by hand