Save your engineers' sleep: best practices for on-call processes

On-call pain is usually a symptom of noisy, low-quality alerts and missing documentation. By treating alerts like code and forcing every change through review, you eliminate ad-hoc modifications and create a single source of truth that engineers can trust.

Ably's approach combines several concrete tactics: alerts are defined in Terraform modules, reviewed like any other code change, and built on percentiles rather than simple averages to surface tail-risk issues. Each alert is paired with a playbook that outlines impact, severity, debugging steps, and remediation, turning a vague page into a clear action plan. Prometheus Alertmanager handles deduplication, silencing, and routing, while the Karma UI gives a readable overview of current alerts.

Operational polish comes from tooling that makes on-call ownership visible and predictable. PagerBeauty surfaces who is on-call for every rotation, and an automated test page verifies the alert pipeline each shift. A lightweight incident management framework defines roles, escalation paths, and post-mortem documentation, reducing confusion and preventing burnout.

The result is a quieter paging signal, faster resolution times, and a happier engineering team that can focus on building value instead of firefighting. Teams that adopt these practices see measurable improvements in reliability, morale, and the ability to scale incident response without adding headcount.