Argo Workflows runs DAGs of containers. It is not CI (Tekton's job here) and not reconciliation (operators' job); it is imperative orchestration with dependencies, retries and parameters, which makes it the right engine for provisioning sequences: validate the request, create the resources, register them elsewhere, notify.

needsmake core

Orientation

competency 3.2 · workflows for self-service provisioning

Run to completion, not converge forever. That distinction is section 3.6's whole decision table, and it is the reason a workflow is the right tool for "stamp this out once when asked" and the wrong tool for "keep this true for the next three years".

Lab note

Workflows installs with the gitops layer and the controller watches the argo and default namespaces. Workflows created elsewhere will sit untouched forever, looking exactly like a broken controller; worth knowing before you spend ten minutes on it.

The model

entrypoint · templates · parameters

A Workflow is a spec with an entrypoint and a list of templates. Templates come in flavours, and picking the right one is most of the authoring skill:

Template typeDoesUse for
containerruns an imageanything with a CLI
scriptinline code with an interpreter; its stdout becomes a resultvalidation, small glue, computed values
resourcecreate/apply/patch/delete a Kubernetes object, optionally waiting on a success conditionthe provisioning workhorse
dagtasks with dependenciesthe shape to default to
stepssequential groups (- parallel within a group, - - serial between groups)simple linear flows
suspendpauses until resumed (manually or after a duration)approval gates

Parameters flow via {{workflow.parameters.x}} and between tasks via outputs ({{tasks.a.outputs.result}}); when gates conditional branches; withItems/withParam fan a template out over a list. Artifacts (S3/MinIO-backed files) move large data between steps; not configured in this lab, but know the word, because "results are small strings, artifacts are files" is the same distinction Tekton draws.

Reusable pieces: WorkflowTemplate (namespaced library, invoked with workflowTemplateRef or submitted from the UI), ClusterWorkflowTemplate (cluster-scoped version), and CronWorkflow (scheduled). Promoting a Workflow to a WorkflowTemplate is what turns a script into a self-service endpoint: it becomes a thing with a name, parameters and a submit button.

WorkflowTemplate "provision-tenant"   ← the product: named, parameterised, submittable
        │ submit(team=team-d)
        ▼
Workflow (run-to-completion)
   dag:
     check ──▶ namespace ──▶ quota ──▶ register
     (script)    (resource)      (resource)   (container)
        │
        └─ each node = one pod, running as spec.serviceAccountName

RBAC is the part that actually fails

and the Argo-specific wrinkle

Workflow pods run as a ServiceAccount, and a resource template creating namespaces needs cluster-scoped rights that default does not have. The exam-shaped skill is wiring SA → Role/ClusterRole → binding → spec.serviceAccountName, and then recognising a Forbidden error inside a workflow node as an RBAC problem rather than a workflow problem.

The one nobody guesses

Argo's executor reports each step's outcome through its own CRD, workflowtaskresults.argoproj.io. Every workflow ServiceAccount therefore needs create,patch on that resource; forget it and even a perfectly correct workflow fails, usually before your actual permission problem surfaces. Seeing that error once is worth an hour of doc reading.

Two related permissions worth knowing exist: the SA also needs get,list,watch on pods to report progress in some configurations, and if you use artifacts, access to the artifact repository Secret. The general rule is the same as section 5.1's: least privilege, then prove it with kubectl auth can-i --as=system:serviceaccount:<ns>:<sa> before you blame the tool.

Diagnosing a failed workflow

kubectl get workflow <name> -o jsonpath='{.status.nodes}' | jq '.[] | {displayName, phase, message}'
kubectl get workflow <name> -o jsonpath='{.status.phase}{"\n"}'
kubectl logs -l workflows.argoproj.io/workflow=<name> --all-containers --tail=50

The node map is the whole diagnosis: each node's phase and message, in one JSON blob. A workflow's failure diagnostics are pod diagnostics plus that one layer.

Workflows as a self-service interface

what makes it a product rather than a script
  • Parameters at submit time: a WorkflowTemplate with typed, defaulted, documented parameters is an API. A Workflow with hard-coded values is a script you happened to run in a pod.
  • Validation before side effects: a first DAG node that rejects bad input (a name that is not a DNS label, a quota over policy) is admission control one layer earlier, and it is much cheaper than half-provisioning and rolling back.
  • Idempotency: a re-run should not double-create. resource templates with action: apply rather than create, or a when guard on an existence check, are the two usual answers. Workflows do not reconcile, so idempotency is your job.
  • Observability: the workflow's own status is the audit trail; ship it somewhere. And verify the artifacts, not the runner: a green workflow that produced nothing is still a failure.
  • Triggering: CronWorkflow for schedules, the API/CLI for portals, or Argo Events (not installed here) for event-driven runs. Backstage templates commonly call one of these underneath, which is how a form becomes infrastructure.
Verify the output, not the runner

Any provisioning task should be graded twice: the workflow reached Succeeded, and the objects it was supposed to create exist with the right content. Exam graders check the second; get in the habit of checking both.

Exercises

tick the dot when its check passes

The self-service story from 3.1, made concrete. First the identity:

kubectl -n default create sa provisioner
kubectl create clusterrole tenant-provisioner --verb=create,get --resource=namespaces,resourcequotas
kubectl create clusterrolebinding tenant-provisioner --clusterrole=tenant-provisioner --serviceaccount=default:provisioner
# Argo's executor reports each step's outcome through its own CRD, so every
# workflow SA needs this too; forget it and even a correct workflow fails:
kubectl -n default create role wf-taskresults --verb=create,patch --resource=workflowtaskresults.argoproj.io
kubectl -n default create rolebinding wf-taskresults --role=wf-taskresults --serviceaccount=default:provisioner

Then the workflow, a two-node DAG using resource templates:

apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata: { generateName: provision-tenant-, namespace: default }
spec:
  entrypoint: provision
  serviceAccountName: provisioner
  arguments: { parameters: [{ name: team, value: team-d }] }
  templates:
    - name: provision
      dag:
        tasks:
          - name: namespace
            template: make-ns
          - name: quota
            template: make-quota
            dependencies: [namespace]
    - name: make-ns
      resource:
        action: create
        manifest: |
          apiVersion: v1
          kind: Namespace
          metadata:
            name: "{{workflow.parameters.team}}"
            labels: { tenant: "{{workflow.parameters.team}}" }
    - name: make-quota
      resource:
        action: create
        manifest: |
          apiVersion: v1
          kind: ResourceQuota
          metadata:
            name: default-quota
            namespace: "{{workflow.parameters.team}}"
          spec:
            hard: { requests.cpu: "1", requests.memory: 2Gi, pods: "10" }

kubectl create -f it (generateName forbids apply), then watch: kubectl get workflow -w.

verify: the workflow reaches Succeeded, and kubectl get ns team-d --show-labels plus kubectl -n team-d get resourcequota show the artifacts. A green workflow that produced nothing would still be a failure; check the output, not the runner.

Rerun with serviceAccountName: default and a new team name. Which rule it names first is instructive: with the bare default SA the executor usually trips over workflowtaskresults (its own reporting channel) before your namespace rule even gets a chance, an Argo-specific wrinkle that looks baffling until you have seen it once.

verify: the workflow fails, and kubectl get workflow <name> -o jsonpath='{.status.nodes}' | jq '.[] | {phase, message}' contains a forbidden message naming a missing verb and resource. The message structure is identical for every RBAC failure you will ever debug.

Submit the same workflow for team-e without editing the file. The UI's submit flow lists WorkflowTemplates, not bare Workflows, so either promote yours to a WorkflowTemplate first (change the kind, drop generateName for a name) and submit it from the UI with the parameter overridden, or stay in the shell and sed 's/team-d/team-e/' workflow.yaml | kubectl create -f -.

verify: two tenants exist, one workflow spec. Parameters-at-submit is what makes a workflow a self-service endpoint rather than a script, and the WorkflowTemplate promotion is exactly how you would productise it.

Add a first DAG node check using a script template that exits non-zero when {{workflow.parameters.team}} doesn't match ^team-[a-z]+$, and make the other nodes depend on it.

verify: a run with team=Team_X fails at check and creates nothing. Input validation before side effects; the same shape as admission, one layer earlier. Clean up the test tenants when done: kubectl delete ns team-d team-e.

Self-check

answer before opening
Which template type creates Kubernetes objects, and what is its idempotency story?

resource. With action: create a re-run fails on AlreadyExists; with action: apply it converges. Workflows do not reconcile, so if the same request may be submitted twice, you design for it: apply semantics, or a guard step.

A workflow node says forbidden. Where do you look, and what is the classic first offender?

The workflow's ServiceAccount and its bindings, not the workflow spec. The classic first offender in Argo is missing create,patch on workflowtaskresults.argoproj.io, the executor's own reporting channel, which fails before your intended resource permission is even exercised.

When is a workflow the wrong tool, and what replaces it?

When the desired state must stay true indefinitely; a workflow runs once and forgets. Use a controller/operator, or a declarative composition engine (Crossplane, kro) whose CR keeps reconciling. Workflows are for sequences with an end.

Two DAG tasks with no dependencies: what happens, and how do you gate one on a condition?

They run in parallel. Gate with dependencies for ordering and when for conditions (evaluated against parameters or a previous task's result). A validation node that everything depends on is the standard "fail before side effects" shape.

What turns a Workflow into a self-service product?

Promotion to a WorkflowTemplate with typed, documented, defaulted parameters, a submit surface (UI, CLI, portal, or a Backstage action), input validation before side effects, and an auditable record of who requested what. Same content, packaged as an interface.

Docs to know your way around

study time, not exam time
  • argo-workflows.readthedocs.io: core concepts, the fields reference for resource templates, DAG examples, and the WorkflowTemplate page.
  • The Workflows UI's built-in examples (submit → examples) are a legitimate crib sheet during practice.
  • Offline: kubectl explain workflow.spec.templates --recursive and argo submit --help.