SRE-final-project/q2-postmortem/postmortem-report.md

Incident Post-mortem: API Service OOMKilled Outage

1. Summary

• Date & Time: [Insert Date]
• Duration: 45 minutes
• Impact: API service was completely unavailable for users, resulting in a 100% error rate (502 Bad Gateway / 503 Service Unavailable) during the incident window.
• Root Cause: Container memory limit was exceeded, causing the Kubernetes/Cloud orchestrator to continuously terminate the pod with an OOMKilled status.

2. Timeline (UTC)

• 10:00 AM: Monitoring alerts triggered for high error rates on the /health endpoint.
• 10:05 AM: On-call engineer acknowledged the alert and started investigation.
• 10:15 AM: Engineer identified that the container was crash-looping with OOMKilled exit code 137.
• 10:25 AM: A temporary mitigation was applied by manually increasing the container memory limit from 512MB to 1024MB.
• 10:35 AM: Service stabilized. Containers remained running without restarts.
• 10:45 AM: Incident marked resolved after 10 minutes of stable metrics.

3. Root Cause Analysis (The "5 Whys")

1. Why did the service go down? The container was repeatedly killed by the orchestrator.
2. Why was it killed? The orchestrator issued an OOMKilled (Out Of Memory) signal.
3. Why did it run out of memory? The application consumed more memory than its allocated limit (512MB).
4. Why did it consume so much memory? An unexpected spike in requests (or a memory leak in a newly deployed feature) caused the application stack to load massive objects into memory simultaneously.
5. Why wasn't this caught or handled? The auto-scaling policy was based solely on CPU, so it didn't spin up new instances to distribute the memory load.

4. Recommendations & Action Items

• Immediate: Keep the memory limit at 1024MB until a thorough memory profiling is completed.
• Short-term: Implement a memory-based auto-scaling rule (Scale out when Memory > 70%).
• Medium-term: Setup early-detection alerts for memory utilization reaching 80% to warn the team before an OOMKilled event occurs.
• Long-term: Profile the application to identify memory bottlenecks or leaks.

5. Auto-Scaling Policy

To prevent repeating this incident, the platform's auto-scaling group must be configured as follows:

• Scale-Out Policy: Add 1 container instance when Average Container Memory > 70% for 2 minutes.
• Scale-In Policy: Remove 1 container instance when Average Container Memory < 40% for 5 minutes.
• CPU Backup Policy: Scale out if Average CPU > 75% for 2 minutes.

6. Early Detection

To detect this issue before the container crashes:

• Configure alerts on the metric container_memory_usage_bytes (or equivalent).
• Warning Alert: Container Memory > 80% (Sustained for > 3 minutes) - triggers Slack/Teams notification.
• Critical Alert: Container Memory > 90% (Sustained for > 2 minutes) - triggers PagerDuty to wake up on-call engineer for immediate manual mitigation.