41 أسطر
2.9 KiB
Markdown
41 أسطر
2.9 KiB
Markdown
# Incident Post-mortem: API Service OOMKilled Outage
|
|
|
|
## 1. Summary
|
|
- **Date & Time:** [Insert Date]
|
|
- **Duration:** 45 minutes
|
|
- **Impact:** API service was completely unavailable for users, resulting in a 100% error rate (502 Bad Gateway / 503 Service Unavailable) during the incident window.
|
|
- **Root Cause:** Container memory limit was exceeded, causing the Kubernetes/Cloud orchestrator to continuously terminate the pod with an `OOMKilled` status.
|
|
|
|
## 2. Timeline (UTC)
|
|
- **10:00 AM:** Monitoring alerts triggered for high error rates on the `/health` endpoint.
|
|
- **10:05 AM:** On-call engineer acknowledged the alert and started investigation.
|
|
- **10:15 AM:** Engineer identified that the container was crash-looping with `OOMKilled` exit code 137.
|
|
- **10:25 AM:** A temporary mitigation was applied by manually increasing the container memory limit from 512MB to 1024MB.
|
|
- **10:35 AM:** Service stabilized. Containers remained running without restarts.
|
|
- **10:45 AM:** Incident marked resolved after 10 minutes of stable metrics.
|
|
|
|
## 3. Root Cause Analysis (The "5 Whys")
|
|
1. **Why did the service go down?** The container was repeatedly killed by the orchestrator.
|
|
2. **Why was it killed?** The orchestrator issued an `OOMKilled` (Out Of Memory) signal.
|
|
3. **Why did it run out of memory?** The application consumed more memory than its allocated limit (512MB).
|
|
4. **Why did it consume so much memory?** An unexpected spike in requests (or a memory leak in a newly deployed feature) caused the application stack to load massive objects into memory simultaneously.
|
|
5. **Why wasn't this caught or handled?** The auto-scaling policy was based solely on CPU, so it didn't spin up new instances to distribute the memory load.
|
|
|
|
## 4. Recommendations & Action Items
|
|
- **Immediate:** Keep the memory limit at 1024MB until a thorough memory profiling is completed.
|
|
- **Short-term:** Implement a memory-based auto-scaling rule (Scale out when Memory > 70%).
|
|
- **Medium-term:** Setup early-detection alerts for memory utilization reaching 80% to warn the team *before* an OOMKilled event occurs.
|
|
- **Long-term:** Profile the application to identify memory bottlenecks or leaks.
|
|
|
|
## 5. Auto-Scaling Policy
|
|
To prevent repeating this incident, the platform's auto-scaling group must be configured as follows:
|
|
- **Scale-Out Policy:** Add 1 container instance when Average Container Memory > 70% for 2 minutes.
|
|
- **Scale-In Policy:** Remove 1 container instance when Average Container Memory < 40% for 5 minutes.
|
|
- **CPU Backup Policy:** Scale out if Average CPU > 75% for 2 minutes.
|
|
|
|
## 6. Early Detection
|
|
To detect this issue before the container crashes:
|
|
- Configure alerts on the metric `container_memory_usage_bytes` (or equivalent).
|
|
- **Warning Alert:** Container Memory > 80% (Sustained for > 3 minutes) - triggers Slack/Teams notification.
|
|
- **Critical Alert:** Container Memory > 90% (Sustained for > 2 minutes) - triggers PagerDuty to wake up on-call engineer for immediate manual mitigation.
|