4.2 Alerting: rules, Alertmanager, routing

Two systems, one handoff, and keeping their jobs separate answers half the questions about them. Prometheus evaluates rules and decides what is true. Alertmanager receives firing alerts and decides who hears about it.

needsmake up obs

Orientation

competency 4.1 · alerting and notification routing

"Where do I configure X" questions are really "which side of the handoff is X" questions. Thresholds, durations and labels: Prometheus. Grouping, routing, silencing, inhibition, receivers: Alertmanager. Get that boundary right and the rest is syntax.

Prometheus                              Alertmanager
 rule groups, evaluated every 30s          receives firing alerts via HTTP
   expr → Inactive → Pending (for:) → Firing  ─────▶  route → group → inhibit → silence → receiver
   labels decide routing                      group_wait / group_interval / repeat_interval
   annotations describe                       webhook · email · Slack · PagerDuty

Rules

PrometheusRule · states · recording rules

A PrometheusRule CRD holds groups of rules. An alerting rule is a PromQL expression plus:

for: how long the expression must hold before Pending becomes Firing. This is your flap filter, and picking it is a real decision: too short and you page on transients, too long and you notice late.
labels: routing material. severity, team, service. Whatever your routes match on must be produced here.
annotations: human material: summary, description, runbook_url. Templated with {{ $labels.x }} and {{ $value }}.
keep_firing_for is the opposite of for: keeps an alert firing briefly after it resolves, to damp flapping resolutions.

Recording rules are the other half of the CRD and are underrated: they precompute an expensive expression into a new series on a schedule (job:http_errors:rate5m, by convention level:metric:operation). Dashboards and alerts then read a cheap series. If a scenario says "this dashboard takes 30 seconds to load", a recording rule is the expected answer.

The same selector story as ServiceMonitors

Rules are only evaluated if they match the Prometheus object's ruleSelector. This lab's is {} (everything); a stock install's is the release label. An unmatched rule produces no evaluation and no error. Read the selector before blaming the rule, and note that this is the third place in domain 4 where a selector silently discards correct configuration.

Alert states, and where each is visible

Inactive → Pending (expression true, for not yet satisfied) → Firing. Pending alerts appear in Prometheus's Alerts page and nowhere else; they have not been sent. That matters when someone asks why nothing reached the receiver "even though the alert is showing". Add evaluation interval + for + group_wait to compute how long a fire genuinely takes; on a stock stack that is easily 2–3 minutes, and knowing to wait is worth a mark.

Add the settings up before you conclude that alerting is broken:

What makes an alert worth having

Symptom over cause. Alert on user-visible failure (error rate, latency, unavailability), not on every internal cause. Cause-based alerts multiply; symptom-based ones stay bounded.
Actionable. If nobody would do anything at 3am, it is a dashboard panel, not a page.
SLO-shaped, if you can. Burn-rate alerting (fast burn on a short window, slow burn on a long one) is the modern form: it pages on error budget consumption instead of arbitrary thresholds. Even knowing the phrase "multi-window multi-burn-rate" signals you have read the SRE material.
Documented. A runbook_url annotation is the cheapest reliability improvement in this section.

Routing

tree · grouping · silence · inhibition

Note the order: routing happens first. The dispatcher matches an alert against the route tree, and the route it lands on decides the group (its group_by) and the timers; only then does the per-group pipeline run inhibition, silences, waiting and de-duplication before notifying. Reading it the other way round leads to conclusions like "the silence should have stopped it from grouping", which is not a thing.

Alertmanager's config is a tree of routes. Each alert enters at the root and descends to the most specific matching route; continue: true lets it match siblings too. A route names a receiver and grouping behaviour:

Setting	Means	Typical
group_by	collapse alerts sharing these labels into one notification	[alertname, namespace]
group_wait	wait before the first send, to batch siblings	30s
group_interval	wait before sending new members of an existing group	5m
repeat_interval	how often to re-notify about an unresolved group	4h–12h
matchers	which alerts take this branch	severity="critical"

Silences mute matching alerts for a time window without touching config; created in the UI or via amtool, and they change notification, never truth: Prometheus still shows the alert firing. Inhibition suppresses alerts when a related, more severe one is already firing (node down inhibits everything on that node), matched by source_matchers, target_matchers and an equal label list. Distinguishing silence (temporary, human, targeted) from inhibition (permanent rule, relationship-based) is a fair exam question.

Verify the rendered truth

In this stack the live config is generated by the operator into a Secret, and the UI's Status page shows the rendered result. The operator also supports namespaced AlertmanagerConfig CRDs that merge into the tree; whether an Alertmanager picks them up depends on its alertmanagerConfigSelector. So verify pickup in the rendered config rather than trusting the apply. That habit transfers to every operator-managed config on the exam.

One special alert to recognise: Watchdog, which fires always, by design. It is a dead-man's switch: an external system watches for it and screams when it stops arriving, which is how you detect that your whole alerting pipeline died. If a question asks why an always-firing alert is a feature, that is the answer.

Exercises

tick the dot when its check passes

Alertmanager has no LoadBalancer here: kubectl -n monitoring port-forward svc/prometheus-kube-prometheus-alertmanager 9093:9093 and browse localhost:9093.

kubectl apply -f - <<'EOF'
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: curriculum-drill
  namespace: monitoring
  labels: { release: prometheus }
spec:
  groups:
    - name: drill
      rules:
        - alert: TooManyExamplePods
          expr: count(kube_pod_info{namespace="default"}) > 0
          for: 1m
          labels: { severity: warning, team: platform }
          annotations:
            summary: "{{ $value }} pods in default"
            description: "Drill alert; fires whenever default has any pods."
EOF

Watch it walk the states: Prometheus UI → Alerts shows Pending, then Firing after the minute.

verify: it appears in the Alertmanager UI with your labels, grouped, and the annotation rendered the live value rather than the template text.

In the Alertmanager UI, create a silence matching alertname=TooManyExamplePods for 2 hours with a comment.

verify: the alert leaves the active view and lands under Silences, while Prometheus still shows it Firing. That split is the lesson: silencing changes notification, never truth.

Open Status in the Alertmanager UI and answer from the rendered config: what is the root receiver, what does group_by collapse on, which special route exists for the Watchdog alert (the stack ships one that fires always, as a dead-man's switch; know why that is a feature).

verify: you can trace where your drill alert's severity: warning lands in the tree and say which single config line you would change to route team: platform alerts to a new webhook receiver.

Deploy a trivial webhook sink (kubectl create deploy sink --image=mendhak/http-https-echo:31 plus a Service), add an AlertmanagerConfig in default routing team: platform to it, and then verify pickup honestly: does the rendered config in the UI now contain the route? If yes, kubectl logs deploy/sink shows the JSON payload when the drill alert fires. Read it once; its structure (groupLabels, commonAnnotations, alerts[]) is what every receiver integration parses.

verify: either the payload arrives, or you can name the selector on the Alertmanager object that would have to change. Diagnosing that honestly is worth as much as the happy path. Clean up: delete the PrometheusRule; the silence expires on its own.

Self-check

answer before opening

An alert shows Firing in Prometheus and nothing arrived. Three candidate causes?

A silence matches it; an inhibition rule suppresses it; or routing sent it to a receiver that is failing (check Alertmanager's own logs and its alertmanager_notifications_failed_total). A fourth, if it only just fired: group_wait has not elapsed.

Your rule applies cleanly and never evaluates. What did you not check?

The Prometheus object's ruleSelector (and its namespace selector). Unmatched rules are ignored silently, the same failure shape as ServiceMonitors, with the same first command: read the selector.

Silence versus inhibition: when do you use each?

Silence: temporary, human-initiated, targeted at a known maintenance or a known-noisy alert, with an expiry and a comment. Inhibition: a standing rule expressing a relationship: when the cause alert fires, suppress its downstream symptoms. Silences are operations; inhibitions are design.

How long, roughly, from "condition becomes true" to "notification sent" on a stock stack?

Up to one evaluation interval (30s) to notice, plus for (say 1–5m), plus group_wait (30s). So a couple of minutes minimum, which is why you wait before declaring an alerting pipeline broken, and why for: 15m on a page-worthy symptom is usually too slow.

What is the Watchdog alert for?

It fires permanently as a dead-man's switch: an external system expects to keep receiving it and alerts when it stops, catching the failure mode where Prometheus or Alertmanager itself dies and therefore cannot alert you about anything. Self-monitoring has to come from outside.

Docs to know your way around

study time, not exam time

prometheus.io: alerting rules, and the Alertmanager configuration page (route and inhibit_rule syntax).
prometheus-operator.dev: PrometheusRule and AlertmanagerConfig CRD references.
sre.google: the SRE workbook chapter on alerting on SLOs, for burn-rate vocabulary.
Offline: amtool config routes show / amtool config routes test if the binary is present, and the Alertmanager UI's Status page.