Prometheus: Writing Alerts That Won’t Get Ignored

Any team that has used Prometheus^[1] long enough has lived the same cycle. Alerts are added enthusiastically, and six months later the on-call channel is flooded with noise. Nobody reads it, and when something serious happens the signal is lost among false positives. The problem is rarely Prometheus: it’s the rule design.

Key takeaways

• Alert on customer-observable symptoms (latency, error rate, saturation), not on internal causes (high CPU, low memory).

• A well-written alert includes a non-trivial for, routing labels, and annotations with summary, description, runbook, and dashboard.

• SLOs with multi-window burn rate drastically reduce unnecessary pages and align alerts with real customer promises.

• The watchdog, an alert that always fires, detects when the alerting system is silent without reason.

• Quarterly review of signal/noise ratio is as important as writing the initial rules.

Symptoms vs. causes: alert on what matters to the user

The most important alert-design rule, defended by Google’s SRE team in the original SRE book^[2], is: alert on symptoms, not causes.

• Symptom: "The 5xx error rate on /api/payments exceeds 1% for 5 minutes."

• Cause: "Pod payments-service-3 is at 95% CPU."

The difference matters because a user doesn’t experience high CPU: they experience slow responses or errors. Alerting on causes produces two simultaneous pathologies:

• False positives: a cause can fire without the user noticing (the service auto-scales and absorbs the spike).

• False negatives: an unforeseen cause can produce a failure with no cause-level alert firing.

A good ruleset starts from symptoms observable from the customer’s perspective (latency, error rate, saturation) and keeps causes as diagnostic dashboards, not as paging alerts.

Anatomy of a well-written alert

A Prometheus rule with multiple windows, complete annotations, and routing labels:

- alert: ApiHighErrorRate
  expr: |
    sum by (service) (
      rate(http_requests_total{status=~"5.."}[5m])
    )
    /
    sum by (service) (
      rate(http_requests_total[5m])
    )
    > 0.01
  for: 10m
  labels:
    severity: page
    team: platform
  annotations:
    summary: "API {{ $labels.service }} error rate above 1%"
    description: |
      Service {{ $labels.service }} has had >1% 5xx error rate for the
      last 10 minutes (current: {{ $value | humanizePercentage }}).
    runbook_url: "https://runbooks.example.com/api-error-rate"
    dashboard_url: "https://grafana.example.com/d/abc/api-overview"

Four elements that must not be missing:

• Non-trivial for. A window of 5 to 15 minutes absorbs transients without excessively delaying response to real incidents.

• Clear routing labels. severity (page / ticket / info) + team let Alertmanager route to different channels and each team receives only what’s theirs.

• Complete annotations. summary (one line), description (context with interpolated values), runbook_url (what to do), dashboard_url (where to look). An alert without a runbook is an invitation to panic.

• Correct PromQL. Use rate() on counters, not increase() directly. Group by the dimensions that matter for routing.

Prometheus logo: the open-source monitoring and alerting system described in this article (Image: Alexander Schwartz (ahus1), Apache License 2.0, via Wikimedia Commons)

SLOs and multi-window burn rate

The pattern that has gained fastest adoption is SLO-based alerts with multi-window, multi-threshold burn rate, popularised by Google SRE and detailed in chapter 5 of the SRE Workbook^[3].

The idea: define an SLO (say, 99.9% success over 30 days, allowing 0.1% error budget). Instead of alerting on absolute error rate, alert when you’re burning the error budget faster than sustainable:

• Burn rate > 14.4x for 1h → critical alarm (you’d consume the month’s budget in 2 days).

• Burn rate > 6x for 6h → serious alarm (you’d consume the budget in 5 days).

• Burn rate > 1x for 24h → trend alarm (you’re on track to spend the budget).

This aligns alerts with real customer promises (the SLO) and drastically reduces unnecessary pages. Sloth^[4] and Pyrra^[5] generate these rules automatically from a declarative SLO definition.

This monitoring infrastructure is especially valuable when combined with kernel-level observability from eBPF: high-level metrics in Prometheus and kernel granularity in eBPF complement each other rather than competing.

The watchdog: an alert that’s always on

A common mistake: silent alerts. Prometheus stops scraping, Alertmanager crashes, or a config error means rules don’t evaluate. No alert fires, but no ping arrives either. Two weeks later you discover your observability has been dead.

The canonical solution: a watchdog alert that is always firing by design:

- alert: Watchdog
  expr: vector(1)
  labels:
    severity: none
  annotations:
    summary: "Prometheus is alive"

Sent to a receiver that expects it every X minutes. If it doesn’t arrive within a threshold, the receiver (Dead Man’s Snitch^[6] or Healthchecks.io^[7], for example) fires its own alert. This turns silence into signal rather than ambiguity.

What to polish quarterly

Alerts aren’t "configure and forget". A useful ritual for on-call teams:

• Quarterly review of top-N pages. Which alerts fired the most? How many led to real action? Ones always acknowledged without action should be removed or tuned.

• Post-mortems with an "alerts" item. Each incident teaches: did the right alert arrive in time? Did irrelevant alerts fire in parallel?

• Testing new alerts in staging. Simulate the symptom before promoting the rule to production.

This continuous-improvement discipline connects with design thinking applied to operations: the best runbooks and alerts are designed from the perspective of the operator under pressure, not the author with full context. And as our guide to installing Traefik with Docker Compose shows, observability starts at the infrastructure layer before reaching business metrics.

Conclusion

Four principles reduce on-call fatigue and improve response to real incidents. Alert on symptoms, base severity on SLOs with burn rate, monitor the alerting system itself with a watchdog, and review the signal/noise ratio quarterly. The difference between a useful on-call channel and an ignored one lies in rule design, not in the quantity of metrics collected.

Frequently asked questions

How long should the `for` clause be on a Prometheus alert?

Between 5 and 15 minutes: a for in that range absorbs transients without excessively delaying the response to real incidents. A trivial for fires on every momentary spike and feeds the noise that eventually gets the on-call channel ignored. Alongside for, the rule needs routing labels (severity with page, ticket or info values, plus team) and annotations with summary, description, runbook_url and dashboard_url; an alert without a runbook is an invitation to panic.

How do I find out that Prometheus or Alertmanager has stopped working if no alert arrives?

With a watchdog alert that is always firing by design: expr: vector(1) with severity: none. It is sent to an external receiver that expects it every X minutes, for example Dead Man's Snitch or Healthchecks.io. If it does not arrive within the threshold, the receiver fires its own alert. That turns silence into signal rather than ambiguity, so you avoid discovering two weeks late that Prometheus stopped scraping, Alertmanager crashed or a config error meant rules were not being evaluated.

Which burn-rate thresholds should I use to alert on an SLO?

For an SLO such as 99.9% success over 30 days (0.1% error budget), the multi-window pattern from Google SRE alerts when you burn the budget faster than sustainable. A burn rate above 14.4x for 1 hour is a critical alarm: you would consume the month's budget in 2 days. Above 6x for 6 hours is a serious alarm (5 days), and above 1x for 24 hours is a trend alarm. Sloth and Pyrra generate these rules automatically from a declarative SLO definition.

Sources

1. Prometheus
2. original SRE book
3. SRE Workbook
4. Sloth
5. Pyrra
6. Dead Man’s Snitch
7. Healthchecks.io