4.3 KiB
Incident Postmortem: Repeated OOMKilled Causing Application Downtime
Incident Summary
Incident Title: Repeated OOMKilled Causing Application Downtime
Date: July 28, 2026
Duration: 45 Minutes
Severity: High
Affected Service: Ghaymah SRE API
Impact
- The application was unavailable for approximately 45 minutes.
- Users experienced failed requests and service interruptions.
- Multiple container restarts occurred due to repeated OOMKilled events.
Timeline
| Time | Event |
|---|---|
| 10:00 | Application deployed successfully |
| 10:08 | Memory usage started increasing rapidly |
| 10:12 | First OOMKilled event occurred |
| 10:13 - 10:40 | Container repeatedly restarted due to memory exhaustion |
| 10:20 | Monitoring system generated alerts |
| 10:25 | SRE team started investigation |
| 10:35 | Memory leak identified |
| 10:42 | Memory limit increased and deployment restarted |
| 10:45 | Service fully recovered |
Root Cause Analysis
The application experienced a memory leak that continuously increased memory consumption.
Once the container exceeded its configured memory limit, Kubernetes terminated it with an OOMKilled event.
Because the underlying issue was not resolved immediately, the container repeatedly restarted, causing approximately 45 minutes of service downtime.
Recommendations
Immediate Actions
- Increase the container memory limit.
- Restart the deployment after applying the fix.
- Monitor memory usage during recovery.
Long-Term Improvements
- Configure Horizontal Pod Autoscaler (HPA).
- Set appropriate CPU and memory requests and limits.
- Enable memory utilization alerts.
- Monitor container restart counts.
- Perform load testing before production deployments.
- Regularly profile the application to detect memory leaks.
Auto-Scaling Policy
Objective
Automatically scale the application based on workload to maintain availability and reduce the risk of resource exhaustion.
| Configuration | Value |
|---|---|
| Minimum Replicas | 2 |
| Maximum Replicas | 10 |
| CPU Target | 70% |
| Memory Target | 75% |
| Scale Up | Add 1–2 replicas after 2 minutes above threshold |
| Scale Down | Remove 1 replica after 10 minutes of low utilization |
Scale-Up Rules
Trigger scaling when:
- CPU usage exceeds 70%.
- Memory usage exceeds 75%.
- Traffic increases significantly.
Action:
- Create additional application replicas.
- Distribute requests using the load balancer.
Scale-Down Rules
Trigger scaling when:
- CPU remains below 30%.
- Memory remains below 40%.
- Low traffic continues for at least 10 minutes.
Action:
- Gradually remove unused replicas while keeping at least two running.
Early Detection Using Monitoring
Monitoring Metrics
Memory Usage
Monitor:
- Container memory usage
- Memory utilization percentage
- Available memory
Alert Rule
Memory usage > 80% for 5 minutes
OOMKilled Events
Monitor:
- Number of OOMKilled events
- Container restart count
Alert Rule
Restart count > 3 within 10 minutes
Container Health
Monitor:
- /health endpoint
- Readiness probe
- Liveness probe
- Pod status
Alert if:
- Health endpoint returns a non-200 status.
- Pods enter CrashLoopBackOff or Pending state.
Application Performance
Monitor:
- HTTP response time
- Request rate
- Error rate (5xx)
- Active requests
Monitoring Dashboard
The dashboard should display:
- Application Status
- CPU Usage
- Memory Usage
- Response Time
- Total Requests
- Container Restarts
- OOMKilled Events
- Running Replicas
Alert Workflow
- Prometheus collects application and container metrics.
- Alertmanager evaluates alert rules.
- Notifications are sent to the SRE team (Email, Slack, or Microsoft Teams).
- Engineers investigate the issue using Grafana dashboards and application logs.
- If auto-scaling is enabled, additional replicas are created automatically while the issue is being investigated.
Conclusion
The outage was caused by excessive memory consumption that resulted in repeated OOMKilled events and continuous container restarts.
Implementing proactive monitoring, alerting, resource limits, and auto-scaling policies will significantly reduce the likelihood and impact of similar incidents in the future.