196 أسطر
4.3 KiB
Markdown
196 أسطر
4.3 KiB
Markdown
# Incident Postmortem: Repeated OOMKilled Causing Application Downtime
|
||
|
||
## Incident Summary
|
||
|
||
**Incident Title:** Repeated OOMKilled Causing Application Downtime
|
||
|
||
**Date:** July 28, 2026
|
||
|
||
**Duration:** 45 Minutes
|
||
|
||
**Severity:** High
|
||
|
||
**Affected Service:** Ghaymah SRE API
|
||
|
||
### Impact
|
||
|
||
- The application was unavailable for approximately 45 minutes.
|
||
- Users experienced failed requests and service interruptions.
|
||
- Multiple container restarts occurred due to repeated OOMKilled events.
|
||
|
||
---
|
||
|
||
# Timeline
|
||
|
||
| Time | Event |
|
||
|------|-------|
|
||
| 10:00 | Application deployed successfully |
|
||
| 10:08 | Memory usage started increasing rapidly |
|
||
| 10:12 | First OOMKilled event occurred |
|
||
| 10:13 - 10:40 | Container repeatedly restarted due to memory exhaustion |
|
||
| 10:20 | Monitoring system generated alerts |
|
||
| 10:25 | SRE team started investigation |
|
||
| 10:35 | Memory leak identified |
|
||
| 10:42 | Memory limit increased and deployment restarted |
|
||
| 10:45 | Service fully recovered |
|
||
|
||
---
|
||
|
||
# Root Cause Analysis
|
||
|
||
The application experienced a memory leak that continuously increased memory consumption.
|
||
|
||
Once the container exceeded its configured memory limit, Kubernetes terminated it with an **OOMKilled** event.
|
||
|
||
Because the underlying issue was not resolved immediately, the container repeatedly restarted, causing approximately 45 minutes of service downtime.
|
||
|
||
---
|
||
|
||
# Recommendations
|
||
|
||
## Immediate Actions
|
||
|
||
- Increase the container memory limit.
|
||
- Restart the deployment after applying the fix.
|
||
- Monitor memory usage during recovery.
|
||
|
||
## Long-Term Improvements
|
||
|
||
- Configure Horizontal Pod Autoscaler (HPA).
|
||
- Set appropriate CPU and memory requests and limits.
|
||
- Enable memory utilization alerts.
|
||
- Monitor container restart counts.
|
||
- Perform load testing before production deployments.
|
||
- Regularly profile the application to detect memory leaks.
|
||
|
||
---
|
||
|
||
# Auto-Scaling Policy
|
||
|
||
## Objective
|
||
|
||
Automatically scale the application based on workload to maintain availability and reduce the risk of resource exhaustion.
|
||
|
||
| Configuration | Value |
|
||
|--------------|-------|
|
||
| Minimum Replicas | 2 |
|
||
| Maximum Replicas | 10 |
|
||
| CPU Target | 70% |
|
||
| Memory Target | 75% |
|
||
| Scale Up | Add 1–2 replicas after 2 minutes above threshold |
|
||
| Scale Down | Remove 1 replica after 10 minutes of low utilization |
|
||
|
||
## Scale-Up Rules
|
||
|
||
Trigger scaling when:
|
||
|
||
- CPU usage exceeds 70%.
|
||
- Memory usage exceeds 75%.
|
||
- Traffic increases significantly.
|
||
|
||
Action:
|
||
|
||
- Create additional application replicas.
|
||
- Distribute requests using the load balancer.
|
||
|
||
## Scale-Down Rules
|
||
|
||
Trigger scaling when:
|
||
|
||
- CPU remains below 30%.
|
||
- Memory remains below 40%.
|
||
- Low traffic continues for at least 10 minutes.
|
||
|
||
Action:
|
||
|
||
- Gradually remove unused replicas while keeping at least two running.
|
||
|
||
---
|
||
|
||
# Early Detection Using Monitoring
|
||
|
||
## Monitoring Metrics
|
||
|
||
### Memory Usage
|
||
|
||
Monitor:
|
||
|
||
- Container memory usage
|
||
- Memory utilization percentage
|
||
- Available memory
|
||
|
||
**Alert Rule**
|
||
|
||
```
|
||
Memory usage > 80% for 5 minutes
|
||
```
|
||
|
||
### OOMKilled Events
|
||
|
||
Monitor:
|
||
|
||
- Number of OOMKilled events
|
||
- Container restart count
|
||
|
||
**Alert Rule**
|
||
|
||
```
|
||
Restart count > 3 within 10 minutes
|
||
```
|
||
|
||
### Container Health
|
||
|
||
Monitor:
|
||
|
||
- /health endpoint
|
||
- Readiness probe
|
||
- Liveness probe
|
||
- Pod status
|
||
|
||
Alert if:
|
||
|
||
- Health endpoint returns a non-200 status.
|
||
- Pods enter CrashLoopBackOff or Pending state.
|
||
|
||
### Application Performance
|
||
|
||
Monitor:
|
||
|
||
- HTTP response time
|
||
- Request rate
|
||
- Error rate (5xx)
|
||
- Active requests
|
||
|
||
---
|
||
|
||
# Monitoring Dashboard
|
||
|
||
The dashboard should display:
|
||
|
||
- Application Status
|
||
- CPU Usage
|
||
- Memory Usage
|
||
- Response Time
|
||
- Total Requests
|
||
- Container Restarts
|
||
- OOMKilled Events
|
||
- Running Replicas
|
||
|
||
---
|
||
|
||
# Alert Workflow
|
||
|
||
1. Prometheus collects application and container metrics.
|
||
2. Alertmanager evaluates alert rules.
|
||
3. Notifications are sent to the SRE team (Email, Slack, or Microsoft Teams).
|
||
4. Engineers investigate the issue using Grafana dashboards and application logs.
|
||
5. If auto-scaling is enabled, additional replicas are created automatically while the issue is being investigated.
|
||
|
||
---
|
||
|
||
# Conclusion
|
||
|
||
The outage was caused by excessive memory consumption that resulted in repeated OOMKilled events and continuous container restarts.
|
||
|
||
Implementing proactive monitoring, alerting, resource limits, and auto-scaling policies will significantly reduce the likelihood and impact of similar incidents in the future.
|