Alertmanager: Routing That Doesn’t Wake Your Team at 3am

A badly configured Alertmanager turns every incident into noise: a single unrouted receiver ends with an ignored Slack channel within a week. This article covers, on Alertmanager 0.27 and Prometheus 2.54, how to design the routing tree, inhibition rules, silences and on-call rotations to curb alert fatigue without losing real incidents.

August 30, 2024 6 min 208 4.2

Methodologies

Prometheus: Writing Alerts That Won’t Get Ignored

To write Prometheus alerts that won't get ignored, alert on customer-observable symptoms (latency, error rate, saturation) instead of internal causes like CPU or memory, define SLOs with multi-window burn rate to scale severity, add a watchdog alert that confirms the system is still alive, and review the signal-to-noise ratio every quarter.

July 1, 2023 5 min 234 3.9