2.2 Argo CD: applications, sync, drift

Argo CD is the tool most likely to be sitting in front of you on the exam, and the Application resource is its entire API. One spec: source, destination, syncPolicy. Everything in the UI is a view over that, and everything that goes wrong is visible in status.

needsmake core

Orientation

competency 2.1 · GitOps workflows

Two hours of Argo CD practice pays back more exam points than any other tool in this curriculum. Know the Application spec cold, know the two status axes, know the CLI, and know where the error text comes from when a sync fails.

The components, because errors are attributable

Component	Job	Errors it produces
repo-server	clones git, renders kustomize/Helm	auth failures, template/render errors, missing paths
application-controller	compares, syncs, reports health	API server rejections, prune/hook behaviour, OutOfSync
api-server	UI, CLI, RBAC, SSO	login and permission errors
redis	cache of rendered manifests and live state	weird staleness after a redis restart

When a task says "the app will not sync", the first fork is: did rendering fail (repo-server) or did applying fail (controller)? The message tells you, and knowing which pod's logs to read halves the time.

The Application spec, field by field that matters

source · destination · syncPolicy

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: demo-staging
  namespace: argocd                      # apps live in the controller's namespace
spec:
  project: default                       # AppProject = the guardrails (see below)
  source:
    repoURL: http://gitea.lab:3000/lab/platform.git
    targetRevision: HEAD                 # branch, tag, or commit SHA
    path: demo-app/overlays/staging      # or chart: + helm: for a chart source
  destination:
    server: https://kubernetes.default.svc   # or name: in-cluster
    namespace: team-a
  syncPolicy:
    automated: { prune: true, selfHeal: true }
    syncOptions:
      - CreateNamespace=true
    retry:
      limit: 3
      backoff: { duration: 5s, factor: 2, maxDuration: 3m }

targetRevision deserves a thought: HEAD or a branch means "whatever moves there", a tag or SHA means immutable. Production apps pinned to a SHA and promoted by bumping it is a legitimate, auditable pattern.

The two status axes, and the mistake of conflating them

Axis	Values	Answers
Sync	Synced · OutOfSync · Unknown	does live match git?
Health	Healthy · Progressing · Degraded · Missing · Suspended · Unknown	are the resources themselves okay?

Synced and Degraded is not a contradiction

Git said run a broken image, and the cluster faithfully runs it broken. Sync is green because the diff is empty; health is red because the Deployment cannot progress. Clicking sync again does nothing at all. The fix is a commit. Recognising this state on sight is worth several minutes on exam day, and section 2.6 opens with it.

Every combination means something different, and each has exactly one right first move:

Health is computed per resource kind by built-in checks (a Deployment is Healthy when its available replicas match, a Service with a LoadBalancer waits for an IP…), and custom kinds get custom Lua health checks, which is how Argo CD reports on CRDs like Rollouts. If a custom resource sits forever in Progressing, either it has no health check or its status conditions never settle.

syncPolicy, decided before you need it

Manual: reports OutOfSync and waits. Correct for production changes that need a human.
automated: syncs when git changes.
automated.selfHeal: true: also reverts live drift, continuously.
automated.prune: true: deletes live resources whose manifests vanished from git. The one that bites: without it, renaming a resource leaves the old one running forever; with it, removing a file deletes production.

Sync options worth memorising

Option	What it fixes
CreateNamespace=true	destination namespace does not exist yet
Replace=true	immutable-field errors; does `kubectl replace` instead of apply
ServerSideApply=true	huge CRDs, field-manager conflicts, last-applied annotation limits
SkipDryRunOnMissingResource=true	CRs whose CRD arrives from elsewhere during the same sync
PrunePropagationPolicy=foreground	ordered deletion of owned resources
ApplyOutOfSyncOnly=true	huge apps where re-applying everything is slow

They can be set per-Application (spec.syncPolicy.syncOptions) or per-resource with the argocd.argoproj.io/sync-options annotation, and knowing both placements is the kind of detail a task quietly depends on.

Ordering: waves and hooks

CRDs before CRs, migrations before rollout

A sync happens in phases, and within the Sync phase in waves. The annotation argocd.argoproj.io/sync-wave: "-1" runs earlier (lower first, default 0); Argo waits for each wave's resources to be healthy before starting the next.

PreSync  ──▶  Sync (wave -1 → 0 → 1 → …)  ──▶  PostSync  ──▶  SyncFail (only on failure)
   │                    │                              │
 db migration Job   CRDs, then CRs, then apps      smoke-test Job

Hooks are ordinary manifests annotated with argocd.argoproj.io/hook: PreSync|Sync|PostSync|SyncFail|PostDelete|Skip, plus hook-delete-policy (HookSucceeded, BeforeHookCreation, HookFailed) so old Jobs do not pile up. The stock examples: a database migration Job in PreSync, a smoke test in PostSync.

The "no matches for kind" fix has two shapes

If you own both the CRD and the CR, put the CRD in wave -1. If the CRD arrives from somewhere you do not control, annotate the CR with SkipDryRunOnMissingResource=true so the pre-apply dry run stops failing. Knowing both is the difference between one fix and a toolbox.

Scale patterns and guardrails

AppProject · app-of-apps · ApplicationSet

AppProject: the part people skip and the exam likes

An AppProject is a policy boundary around a set of Applications: which sourceRepos they may deploy from, which destinations (cluster + namespace pairs) they may target, which cluster-scoped and namespaced resource kinds are allowed or denied, plus per-project RBAC roles and sync windows (time ranges when syncing is permitted or blocked). In a multi-tenant platform this is how you let teams own Applications without letting them deploy a ClusterRoleBinding into kube-system. If a task says "team X must only deploy to namespace Y from repo Z", the answer is an AppProject, not a Role.

app-of-apps vs ApplicationSet

app-of-apps: one Application whose manifests are other Applications. Simple, explicit, reviewable; you write each child by hand. Great for bootstrapping a cluster's platform layer in a defined order (with waves).
ApplicationSet: a template plus generators that stamp out Applications. Generators to recognise: list, clusters, git (directories or files in a repo), scmProvider (every repo in an org matching criteria), pullRequest (ephemeral preview environments), and matrix/merge to combine them. goTemplate: true switches the template engine to Go templating with sprig functions.

For the exam, be able to read a generator block and predict exactly which Applications will exist. That is the testable skill, and it is also what makes the lab's two ApplicationSets legible: examples/argocd-appset.yaml generates one app per overlay directory found in git, and the Backstage golden path (section 3.6) generates apps from an SCM generator over the Gitea services org.

Where does the workload actually land?

An ApplicationSet template sets destination.namespace per generated app, but an explicit namespace: inside a manifest (or a kustomization) always wins. So the lab's overlays land in team-a/team-b regardless of what the template's destination says. Predicting the winner between those two settings is exactly the kind of thing a task hinges on.

Other clusters

destination.server: https://kubernetes.default.svc (or name: in-cluster) is the local cluster. Any other cluster has to be registered first: argocd cluster add <context> creates a ServiceAccount there and stores its credentials in a Secret in argocd labelled argocd.argoproj.io/secret-type: cluster. That Secret is the thing an ApplicationSet's clusters generator iterates, which is why generator-driven fan-out and "deploy this to the second cluster" are the same mechanism. argocd cluster list tells you what is registered; the AppProject's destinations list decides who may target it.

Diffing quirks worth one line each

ignoreDifferences (by group/kind/jsonPointers or a jq path) is how you stop a mutating webhook or an HPA-managed replica count from showing permanent drift. argocd.argoproj.io/compare-options: IgnoreExtraneous hides resources Argo did not create. Both exist because "OutOfSync forever on a field nobody edits" is a real and common state.

CLI, because the UI wastes exam time

argocd login <server> --username admin \
  --password $(kubectl -n argocd get secret argocd-initial-admin-secret -o jsonpath='{.data.password}' | base64 -d) --insecure
argocd app list
argocd app get demo-staging
argocd app diff demo-staging          # what would change
argocd app sync demo-staging          # --prune, --force, --resource
argocd app history demo-staging       # and: argocd app rollback demo-staging <id>
argocd app set demo-staging --sync-policy automated --self-heal

argocd app get --refresh forces a re-comparison against git; --hard-refresh also drops the manifest cache, which is what you reach for when the repo changed but Argo insists it did not.

Exercises

tick the dot when its check passes

Apply the example and predict its output before looking:

kubectl apply -f examples/argocd-appset.yaml
kubectl -n argocd get applications
kubectl -n team-a get deploy staging-demo && kubectl -n team-b get deploy prod-demo

Now the nuance that makes this a good exercise: the ApplicationSet template sets destination.namespace to the directory basename, but the overlays pin namespace: team-a and team-b in their kustomizations, and an explicit namespace in a manifest always wins over the Application's destination default. So the workloads land in the tenant namespaces, prefixed staging- and prod-. If an app is stuck, read its conditions: kubectl -n argocd get app demo-staging -o jsonpath='{.status.conditions}' | jq.

verify: demo-staging and demo-prod Applications exist (one per overlay directory in the platform repo), each Synced/Healthy.

Your replica change from the fundamentals section is already in the platform repo, so demo-staging picks it up on the next poll (up to ~3 min) or immediately with argocd app sync demo-staging.

verify: kubectl -n team-a get deploy staging-demo -o jsonpath='{.spec.replicas}' matches your commit, and argocd app history demo-staging shows the revision.

The appset template sets automated + selfHeal + prune:

kubectl -n team-a scale deploy staging-demo --replicas=5
kubectl -n team-a get deploy staging-demo -w

Then prove you understand prune the safe way: the service manifest lives in demo-app/base/, so drop service.yaml from the base kustomization's resources list, push, and watch the live Service disappear from both tenants (a base edit hits every overlay, which is its own lesson in blast radius). Revert the commit and watch them return.

verify: replicas snap back to the git value within seconds, argocd app get demo-staging never settles OutOfSync, and the Service round-trips with the commit. Git history is your undo; that is the pitch, demonstrated.

Add to the staging overlay a manifest for a CR whose CRD does not exist (any made-up kind), push, and read the sync error. Then fix it properly for the case where you control both: put the CRD in the same overlay with argocd.argoproj.io/sync-wave: "-1" and the CR at wave 0. The other tool for this job is the argocd.argoproj.io/sync-options: SkipDryRunOnMissingResource=true annotation, for when the CRD arrives from somewhere you don't control.

verify: sync succeeds and kubectl get <your-kind> returns the CR. Clean up the overlay afterwards; you'll reuse it.

Push an image tag that doesn't exist (newTag: does-not-exist in the staging kustomization), sync, and read the two statuses side by side: argocd app get demo-staging | head -20. One timing note so you don't misread it: health shows Progressing for up to ten minutes (the Deployment's progressDeadlineSeconds) before flipping to Degraded, while the pod's ImagePullBackOff is visible in events immediately; Progressing-with-a-broken-pod already tells you everything.

verify: Sync status Synced, health on its way to Degraded, and you can articulate why clicking sync again is useless here. Revert the commit. This exact confusion is section 2.6's opening scenario.

Self-check

answer before opening

An app is Synced/Degraded. What are you allowed to conclude, and what is the fix?

Live matches git, so Argo CD has done its job; the desired state itself is broken (bad image, impossible resources, missing config key). The fix is a commit. Re-syncing changes nothing because the diff is already empty.

A sync fails with "field is immutable". Two ways forward?

Set Replace=true (per app or per resource) so the sync uses kubectl replace instead of apply, or delete the resource and let Argo recreate it; Force=true combines the two. Choose deliberately: replace is not a free pass, since a Service manifest that omits clusterIP is still rejected as immutable, and a shrinking PVC is not going to be saved by either option.

What is an AppProject for, in one sentence, and name two things it constrains?

It is the guardrail around a set of Applications: permitted source repos and permitted destinations (cluster + namespace), plus allowed/denied resource kinds, per-project RBAC roles and sync windows. It is the answer to "let this team self-serve deployments without letting them deploy anything anywhere".

You are handed an ApplicationSet with a git directory generator over apps/*/overlays/*. How many Applications will exist?

One per matching directory in the repo at the current revision, so count the directories, then check the template for a name expression that could collide (two identical names silently overwrite). Predicting the exact set from the generator block is the testable skill; kubectl -n argocd get applications then confirms it.

An app shows OutOfSync forever on a field no human edits. What is happening and what fixes it?

Something in-cluster mutates the resource after apply: a mutating webhook, an HPA writing replicas, a defaulting controller. Fix with ignoreDifferences for that path (or stop managing the field). This is a configuration decision, not a bug to chase.

You changed the repo but Argo insists nothing changed. Sequence of moves?

argocd app get --refresh first (recompare), then --hard-refresh (drop the manifest cache), then check the app is on the branch you pushed to (targetRevision), then check repo-server logs for auth or render errors. Cheapest first, always.

Docs to know your way around

study time, not exam time

argo-cd.readthedocs.io: Application spec reference, sync options, sync waves and hooks, AppProject, ApplicationSet generators.
Offline: argocd app --help covers most of what the docs would, and kubectl explain application.spec --recursive works because the Application is just a CRD.