(A few) Ops Lessons We All Learn The Hard Way

Overview
A blog post that shares a candid collection of operational lessons learned the hard way. It highlights common pitfalls in monitoring, incident handling, tooling, and organizational practices, offering practical insights for engineers and leaders.

Key Takeaways

• Email is the worst monitoring and alerting mechanism except for all the others.
• Absence of a signal is itself a signal.
• The severity of an incident is measured by the number of rules broken in resolving it.
• Self-signed certificates proliferate without proper monitoring, leading to fragile setups.
• Turning things off and on again is often a reasonable fix.
• Post-mortem follow-up tasks not picked up within a week rarely get completed.
• Containers create at least as many problems as they solve.

Who Would Benefit

• Site reliability engineers
• Engineering managers
• Technical leads
• Incident response teams
• DevOps practitioners

Frameworks and Methodologies

• Incident Management Process
• Post-mortem Follow-up Workflow
• Monitoring and Alerting Best Practices