Progressive delivery is a Deployment with a spine: the new version goes to a slice of traffic, something measures whether it is worse, and the rollout continues or reverses based on evidence instead of hope.
make core obsmake meshOrientation
Two strategies to know cold, and one distinction that separates a real answer from a memorised one: what actually shifts the traffic.
| Strategy | Shape | Rollback | Costs |
|---|---|---|---|
| Canary | shift traffic in steps, measure between steps | shift back; only the canary slice was exposed | slow; needs metrics worth trusting |
| Blue-green | run both versions full size, cut over at once | instant: flip the selector back | double capacity for the window |
| Rolling (baseline) | replace pods gradually, no analysis | roll forward or undo, no traffic control | what you already had |
| A/B or shadow | route by header/cookie, or mirror traffic | n/a: no user impact for shadow | needs L7 routing (mesh or gateway) |
Argo Rollouts without a traffic provider approximates a 20% canary by running 20% of the replicas: the split is statistical, granularity is limited by replica count, and every client is load-balanced by the Service. With a traffic provider (Istio, Gateway API, NGINX, ALB) or with Flagger driving a mesh, the split is a route weight: exact, independent of replica count, and able to key on headers. That sentence is the compare-and-contrast this section exists for.
Argo Rollouts
A Rollout is a drop-in replacement for a Deployment (same pod template, same selector semantics) whose spec.strategy is richer. It owns ReplicaSets exactly like a Deployment does (the stable one and the canary one), which is why kubectl get rs during a rollout is so legible.
strategy:
canary:
steps:
- setWeight: 20
- pause: { duration: 60s } # omit duration → wait for a human promote
- analysis: # run an AnalysisTemplate, act on the result
templates: [{ templateName: success-rate }]
- setWeight: 50
- pause: {}
# optional: canaryService / stableService / trafficRouting for real weightsThe step list is the whole grammar: setWeight, pause, analysis, plus setCanaryScale and experiment steps. Blue-green swaps that for two Services and a cutover:
strategy:
blueGreen:
activeService: demo-active
previewService: demo-preview
autoPromotionEnabled: false # wait for `promote`
scaleDownDelaySeconds: 30 # keep the old RS warm for instant rollbackAnalysisTemplate is the measurement half: a list of metrics, each with a provider (Prometheus, Datadog, a Job, a web request), an interval, a successCondition or failureCondition, plus failureLimit, count and inconclusiveLimit. An AnalysisRun is one execution of it, and its status carries every measurement, which is where you look when a rollout aborts and you want to know what number killed it. Analyses can run as a rollout step, as a background analysis for the whole rollout, or as a pre-promotion/post-promotion gate in blue-green.
If the metric provider returns an error (bad query, missing series, unreachable Prometheus), the measurement is Error, and errors count against failureLimit just like failures do. So a canary can abort because your PromQL was wrong, not because the new version was. Read the AnalysisRun's measurements before blaming the release, and note that this is faithful to production life, where a broken metrics pipeline blocking a rollout is a feature, not a bug.
Step through a rollout and watch the difference the traffic provider makes:
The verbs, via the kubectl plugin
kubectl argo rollouts get rollout demo -n team-a --watch # the best mental-model builder in domain 2
kubectl argo rollouts set image demo web=<image> -n team-a
kubectl argo rollouts promote demo -n team-a # advance past a pause; --full skips remaining steps
kubectl argo rollouts abort demo -n team-a # stop and shift back to stable
kubectl argo rollouts undo demo -n team-a # roll the spec back to the previous revision
kubectl argo rollouts status demo -n team-a --timeout 60s # scriptable, exits non-zero on failureabort stops the rollout and sends all traffic back to the stable ReplicaSet, but spec still asks for the new image, so status shows Degraded and it will try again if you touch it. undo changes the spec back. "Safely roll back" on an exam means both: abort to stop the bleeding, undo to make the desired state honest. And in a GitOps world, undo's real form is a revert commit, or Argo CD will just re-apply the bad image.
Flagger, for contrast
Flagger inverts the authoring model. You keep authoring a plain Deployment; Flagger's Canary custom resource generates everything else: a <name>-primary Deployment that actually serves, the <name>, <name>-primary and <name>-canary Services, and the mesh routing objects, then shifts real traffic while running its analysis. You never write a Rollout.
kubectl get deploy carefully during a Flagger task
Between rollouts, Flagger scales your original Deployment to zero and serves from the primary copy; your Deployment is the template, not the workload. Seeing podinfo 0/0 next to podinfo-primary 2/2 is the system working, and mistaking it for a broken deployment is the standard first-time reaction.
| Argo Rollouts | Flagger | |
|---|---|---|
| What you author | a Rollout (replaces the Deployment) | a Canary next to an untouched Deployment |
| Traffic control | replica ratio, or a traffic provider | mesh/gateway route weights, always |
| Analysis | AnalysisTemplate + AnalysisRun | analysis.metrics + webhooks (load test, acceptance, confirm-rollout) |
| Promotion | steps, manual promote | automatic once thresholds hold for N intervals |
| Rollback | abort / undo | automatic on threshold failures; the Canary reports Failed |
Flagger's webhooks deserve a name each because they are how it does things metrics cannot: confirm-rollout (a gate before starting), pre-rollout (acceptance test against the canary), rollout (during each step, typically a load generator), confirm-promotion and post-rollout. A canary with no traffic produces no metrics, so the load-test webhook is not decoration; it is what makes the analysis meaningful in a lab.
The mesh cluster runs no Prometheus, and make mesh wires Flagger at a metrics address that does not exist, so the built-in request-success-rate check can never pass there. Build the analysis from webhooks only (a load-test webhook against flagger-loadtester.test, plus a pre-rollout acceptance check if you want a gate). The mechanics you are practising (Canary spec, generated objects, weight progression, events) are entirely real.
How to think about it on exam day
- Pick the strategy from the constraint. Cannot afford double capacity → canary. Need instant rollback and a clean cutover → blue-green. Need to compare behaviour per user segment → A/B with header routing, which requires L7.
- Stateful and schema changes break both. Two versions run at once, so the database must tolerate both. Expand-and-contract migrations are the standard answer, and saying it unprompted signals seniority.
- Choose metrics the user feels. Success rate and latency percentiles over the canary's own series; not CPU, not pod restarts. And make sure the query selects only the canary: an analysis that accidentally measures the stable version will happily promote a broken release.
- Enough traffic to be significant. With three requests a minute, a 20% canary measures noise. Either drive load (Flagger's loadtester, or your own) or lengthen the interval.
- Where GitOps meets this. The rollout object lives in git like everything else; the promotion decision does not. Argo CD syncing a Rollout is fine, but if you
abortwithout reverting git, the next sync re-applies the bad image. Consistency of desired state beats cleverness.
Exercises
kubectl apply -f examples/rollouts/canary.yaml
kubectl argo rollouts get rollout demo -n team-a --watchFirst rollout of a new Rollout goes straight to healthy (nothing to compare against). Now change the image to trigger a real canary, in a second terminal:
kubectl argo rollouts set image demo web=ghcr.io/nginxinc/nginx-unprivileged:1.28-alpine -n team-aCheck the analysis itself: kubectl -n team-a get analysisrun and read one with -o yaml; the measured value and the success condition are both in status. If the analysis errors because the demo app exposes no http_requests_total, that is faithful to production life; read the AnalysisRun error, then either drive traffic that produces the metric or loosen the query, and understand that an erroring analysis is treated per failureLimit too.
Trigger another image change, and while it pauses: kubectl argo rollouts abort demo -n team-a. Then kubectl argo rollouts undo demo -n team-a and confirm Healthy.
Rewrite the Rollout: strategy blueGreen, two Services (demo-active, demo-preview, both selecting app: demo), autoPromotionEnabled: false. Push a new image and inspect both Services' spec.selector before and after kubectl argo rollouts promote demo -n team-a.
Flagger is on the exam's tool list, so this one is not optional. On kind-mesh (kubectx kind-mesh): deploy podinfo and Flagger's loadtester (both from Flagger's podinfo tutorial manifests), then write a Canary CR targeting the podinfo Deployment with provider: istio. Build the analysis from webhooks only: a load-test webhook against the loadtester (http://flagger-loadtester.test/), and if you want a gate, a pre-rollout acceptance webhook. Then bump the podinfo image and watch kubectl describe canary podinfo events walk the weights up.
kubectl get canary -A), and while it runs, kubectl get virtualservice podinfo -o yaml shows Flagger moving real route weights between primary and canary. If the Canary sticks in Progressing, its events name the failing check, which is precisely the diagnostic loop a Flagger exam task would hand you.Self-check
You set setWeight: 10 with 3 replicas and no traffic provider. What actually happens?
Rollouts rounds to whole pods: it runs one canary pod alongside the stable set, so the real split is nearer 25–33%, not 10%. Exact weights require a traffic provider (Istio, Gateway API, NGINX) or more replicas. Saying that out loud is the difference between reciting the field and understanding it.
A canary aborted and the AnalysisRun shows measurements in phase Error. What do you check?
The metric query and its provider: wrong series name, no data yet, unreachable Prometheus, or a selector that matches nothing. Errors count toward failureLimit, so a broken query aborts releases exactly like a bad release does; check the measurement message before touching the application.
Blue-green with autoPromotionEnabled: false: which Service points where, before and after promote?
Before: active selects the old ReplicaSet's pod-template-hash, preview selects the new one. After: both select the new hash, and the old ReplicaSet lingers for scaleDownDelaySeconds so rollback is a selector flip away. Narrating those selectors is the proof you understand it.
Argo CD manages your Rollout. You abort a bad canary. What happens next, and what should you do?
The Rollout spec in git still names the bad image, so the next sync re-applies it and the rollout starts again. Revert the commit (or pin the tag back): abort is an operational stop, git is the desired state. Same lesson as any manual fix under GitOps.
Name two things that make progressive delivery unsafe regardless of tooling.
Database or API changes that both versions cannot tolerate simultaneously (fix with expand-and-contract migrations and backwards-compatible contracts), and metrics too sparse or too coarse to detect harm within the canary window. A third honourable mention: analysis that measures the stable version by accident.
Docs to know your way around
- argo-rollouts.readthedocs.io: canary and blueGreen strategy references, AnalysisTemplate spec, the kubectl plugin page, traffic-router support matrix.
- flagger.app: the Istio canary tutorial and the webhook reference.
- Offline:
kubectl argo rollouts --help,kubectl explain rollout.spec.strategy.canary --recursive,kubectl explain canary.spec.analysis.