GitOps is one idea applied ruthlessly: git holds the desired state, a controller in the cluster pulls it and reconciles continuously, and nobody deploys by pushing manifests at the API server from a laptop or a CI job. Everything else in this domain (Argo CD, Flux, promotion, drift handling) is machinery for that one idea.
make coreOrientation
Domain 2 is the biggest slice of the exam and the most mechanical. The tasks are things like "make this change reach the cluster through git", "this app is OutOfSync, fix it", "pause reconciliation while I operate". All of them assume the model in this section is already automatic for you.
- Declarative: the system's desired state is expressed declaratively.
- Versioned and immutable: desired state is stored so it enforces immutability and versioning, and retains a complete history.
- Pulled automatically: software agents pull the desired state declarations from the source.
- Continuously reconciled: agents continuously observe actual state and attempt to apply the desired state.
Graders like these words. Learn the four nouns (declarative, versioned/immutable, pulled, reconciled) and you can reconstruct the sentences.
The concepts questions hang off
Reconciliation is a loop, not an event
The controller compares live state with git on an interval and on webhooks. So a change made with kubectl edit does not fail; it gets reverted on the next loop if self-heal is on, or reported as drift if not. "Why did my manual fix disappear" is the canonical symptom, and "OutOfSync" is the canonical report. The loop also means recovery is free: delete the whole namespace and the next reconcile rebuilds it, which is the disaster-recovery pitch in one sentence.
┌──────────── git (desired) ────────────┐
│ demo-app/overlays/staging/… │
└────────────────┬──────────────────────┘
│ poll (~3 min) or webhook
▼
controller in the cluster ── compares ──▶ diff
│ │
automated? │ yes → apply selfHeal? │ live drift → revert
│ no → report OutOfSync prune? │ removed from git → delete live
▼
┌──────────── cluster (live) ───────────┐
└───────────────────────────────────────┘
Poke the loop. Commit a change, drift the cluster by hand, delete a manifest from git, then reconcile and see what each setting does:
Push vs pull, in two sentences
CI pushing manifests needs cluster credentials in the CI system, and only knows the state at deploy time. A pull-based agent keeps credentials in-cluster, never exposes them outward, and never stops comparing. That second property, continuous comparison, is why "GitOps" is not just "CI that runs kubectl".
CI builds and tests artifacts and, at most, opens a pull request that changes an image tag. CD is the in-cluster agent reconciling git. If a scenario describes a Jenkins job running kubectl apply against production, the expected critique is: credentials outside the cluster, no drift detection, no continuous reconciliation, and no single source of truth. Say all four.
Repo design, the part people under-prepare
This lab seeds two Gitea repos that model the standard split: platform (cluster-level config: policies, tenants, the app-of-apps) and demo-app (one workload). The split matters because the two have different reviewers, different blast radius and different cadence.
| Choice | Common form | Why |
|---|---|---|
| App source vs config | separate repos | a config change should not rebuild an image, and a code merge should not deploy by accident |
| Environments | directories, not branches | long-lived env branches drift and merges fight; directories diff cleanly and promote by copy |
| Many apps | monorepo with per-app paths, or one repo per app plus a platform repo | monorepo eases atomic cross-app changes; per-app eases ownership and access control |
| Promotion | change flows staging → prod as a PR that bumps an image tag or a patch | the diff is the change record; approvals hang off it naturally |
| Rendered manifests | CI renders kustomize/Helm output to a branch the agent watches | reviewers see final YAML, not template intent; costs you a pipeline |
In this lab, environments are directories: demo-app/overlays/staging and overlays/prod inside the platform repo, and promotion is a change flowing from one directory to the next, usually as an image tag bump or a kustomize patch. Branches-per-environment exists in the wild and the received wisdom is to avoid it, for exactly the reason above.
Templating, both flavours, because the exam can hand you either
Kustomize layers patches over a base with no templating language: resources, patches (strategic-merge or JSON6902), images, namePrefix, commonLabels, configMapGenerator. What you see is YAML all the way down, and kustomize build shows you exactly what will be applied. Helm renders Go templates from values, giving conditionals and loops at the cost of "the manifest does not exist until something renders it". Argo CD and Flux both consume both, and both let you post-render one with the other.
Helm is also a release manager, not only a renderer, and a task can hand you that side of it: helm list -A (releases and revisions), helm upgrade --install --atomic --wait (roll back automatically if it does not become ready), helm history / helm rollback <rel> <rev>, helm get values --revision and helm diff if the plugin is there. Release state lives in Secrets of type helm.sh/release.v1 in the release namespace, which is how you discover what a previous operator did, and why a Flux HelmRelease can adopt or fight an existing release.
Never push an overlay you have not built locally. kustomize build <dir> (or kubectl kustomize) and helm template are the two commands that turn "I think this renders right" into "I know". They also work offline in the exam terminal.
Secrets, the honest caveat
Plain Secrets cannot live in git; base64 is not encryption. The three sanctioned answers are Sealed Secrets (encrypt for git, only the in-cluster controller can decrypt), External Secrets Operator (git holds references, an external store holds truth) and SOPS (encrypted values in git, decrypted by the agent; Flux has native support). Section 5.1 does the mechanics; here, just know that "how do secrets work in GitOps" has a real answer and that "commit it base64-encoded" is not it.
Failure modes to recognise on sight
- Two controllers, one resource. Argo CD and Flux both managing the same manifest is an infinite reconcile war. In this lab they deliberately own different paths, and that separation is the lesson.
- Manual fix disappears. Self-heal did its job. The fix belongs in git; if you truly need a live change, suspend reconciliation first (Flux) or disable auto-sync (Argo), do the surgery, then fold it back.
- Prune surprises. Rename a resource in git without prune and the old one lives forever; with prune, deleting a file deletes production. Know which behaviour you have before you push.
- Immutable fields. Service
clusterIP, selector labels, PVC shrink: apply fails with an API server rejection quoting the field. The GitOps answer is delete-and-recreate or a replace-style sync option, not editing live. - Stale but green. Suspended reconciliation, a lost webhook, a long interval. Everything reports healthy and nothing is current. Always check "when did this last sync" before believing a green dashboard.
Exercises
Never push an overlay you haven't built locally:
kustomize build examples/demo-app/overlays/staging
kustomize build examples/demo-app/overlays/prod | grep -E 'replicas|name:'The platform repo in Gitea carries a copy of examples/. Clone it, change the staging replica count, push:
source lab.env # for GITEA_PASS; anonymous clone works, push needs credentials
git clone "http://lab:${GITEA_PASS}@gitea.lab:3000/lab/platform.git" /tmp/platform && cd /tmp/platform
# edit demo-app/overlays/staging/kustomization.yaml: set replicas, or add a patch
git commit -am "staging: 3 replicas" && git pushBoth controllers are already connected to Gitea:
kubectl -n argocd get secret gitea-repo -o jsonpath='{.data.url}' | base64 -d; echo
flux get sources gitgitea.lab and the Flux GitRepository shows Ready True with a commit SHA. Now you know the truth both engines reconcile from, and it is the same repo you just pushed to.With nothing managing demo-app yet, apply it manually (kubectl apply -k examples/demo-app/overlays/staging -n default after creating the namespace it wants, or just note the rendered namespace). Scale it by hand. Ask: who notices? Nobody, and that is life before GitOps.
Self-check
State the four OpenGitOps principles without looking.
Declarative; versioned and immutable; pulled automatically by agents; continuously reconciled. If you can only remember three, the one people drop is "versioned and immutable", which is also the one that carries the audit and rollback story.
Why are environment branches discouraged?
Long-lived branches accumulate divergent changes, so promotion becomes a merge with conflicts and cherry-picks rather than a reviewable diff, and the environments drift structurally rather than only in the values you meant to differ. Directories keep the difference explicit and diffable.
A colleague fixes production with kubectl edit during an incident. What happens next, and what should they have done?
Self-heal reverts it at the next reconcile (or it is reported as drift). The sanctioned move is to suspend/disable reconciliation for that app, apply the emergency change, then land the same change in git and resume, so the fix survives and the history records it.
What does prune actually change, and what is the failure mode in each setting?
Prune deletes live resources whose manifests disappeared from git. Off: renamed or removed resources linger forever as invisible cruft. On: deleting a file deletes the running thing, so a careless refactor is a production outage. Neither is safe by default; the safety comes from knowing which one you have.
Where do secrets live in a GitOps repo?
Encrypted (Sealed Secrets or SOPS) or by reference (External Secrets pointing at a real store). Never plain, never base64-only. The distinction to state: Sealed Secrets/SOPS keep ciphertext in git; ESO keeps only a pointer and leaves truth in the external store.
Docs to know your way around
- opengitops.dev: the four principles, in exactly the form graders like.
- kustomize.io: bases and overlays, patches, and the transformer list.
- argo-cd.readthedocs.io and fluxcd.io: each has a "core concepts" page worth ten minutes before the tool sections.
- Offline:
kubectl kustomize <dir>,helm template,git log --oneline: your desired state is a repo, so ordinary git tooling is half the diagnosis.