122 أسطر
3.1 KiB
Markdown
122 أسطر
3.1 KiB
Markdown
# Incident Postmortem Report
|
|
## Application Outage Due to Repeated OOMKilled Events
|
|
|
|
## 1. Incident Overview
|
|
Incident ID: INC-2026-001
|
|
Severity: SEV-2
|
|
Duration: 45 minutes
|
|
Service: Ghaymah Hosted Application
|
|
Status: Resolved
|
|
|
|
The application experienced repeated downtime caused by Kubernetes OOMKilled events.
|
|
The container exceeded its allocated memory limit, causing Kubernetes to terminate
|
|
and restart the container repeatedly.
|
|
|
|
|
|
## 2. Executive Summary
|
|
On July 28, 2026, the application experienced intermittent availability issues
|
|
for approximately 45 minutes.
|
|
|
|
The root cause was excessive memory consumption inside the application container.
|
|
Once the container exceeded its configured memory limit, Kubernetes terminated
|
|
the container with an OOMKilled event.
|
|
|
|
The service recovered after adjusting resource configuration and implementing
|
|
a better scaling strategy.
|
|
|
|
## 3. Impact
|
|
During the incident:
|
|
- Users experienced 502/503 errors.
|
|
- API requests failed intermittently.
|
|
- Application availability decreased for 45 minutes.
|
|
- No data loss occurred because the application was stateless.
|
|
|
|
|
|
## 4. Detection
|
|
The incident was detected through:
|
|
- Kubernetes container restart events.
|
|
- OOMKilled status reported by the platform.
|
|
- Increased application response latency.
|
|
- Monitoring alerts triggered by high memory utilization.
|
|
|
|
## 5. Timeline
|
|
| Time | Event |
|
|
|---|---|
|
|
| 10:00 | Traffic increased above normal level |
|
|
| 10:02 | Container memory usage exceeded 90% |
|
|
| 10:04 | Kubernetes terminated container (OOMKilled) |
|
|
| 10:05 | New container instance started |
|
|
| 10:10 | Memory increased again and container crashed |
|
|
| 10:45 | Service stabilized after mitigation |
|
|
|
|
## 6. Root Cause Analysis
|
|
|
|
### Immediate Cause
|
|
The container exceeded its configured memory limit.
|
|
|
|
### Root Cause
|
|
The application was running with insufficient memory resources
|
|
and no horizontal scaling mechanism.
|
|
Possible contributing factors:
|
|
- Memory leak inside the application.
|
|
- Missing memory-based autoscaling.
|
|
- No early warning alerts.
|
|
- Single container handling all traffic.
|
|
|
|
## 7. Resolution
|
|
The incident was resolved by:
|
|
|
|
- Increasing container memory allocation.
|
|
- Restarting unhealthy workloads.
|
|
- Reviewing application memory usage.
|
|
- Preparing an autoscaling policy.
|
|
|
|
## 8. Preventive Actions
|
|
| Action | Priority |
|
|
|---|---|
|
|
| Enable Horizontal Pod Autoscaler | High |
|
|
| Add memory utilization alerts | High |
|
|
| Perform memory profiling | Medium |
|
|
| Improve health checks | Medium |
|
|
| Add resource limits based on metrics | High |
|
|
|
|
|
|
autoscaling:
|
|
minReplicas: 2
|
|
maxReplicas: 8
|
|
|
|
metrics:
|
|
memory:
|
|
targetUtilization: 70
|
|
|
|
cpu:
|
|
targetUtilization: 75
|
|
|
|
behavior:
|
|
scaleUp:
|
|
stabilizationWindowSeconds: 60
|
|
|
|
scaleDown:
|
|
stabilizationWindowSeconds: 300
|
|
|
|
resources:
|
|
requests:
|
|
memory: 256Mi
|
|
cpu: 500m
|
|
|
|
limits:
|
|
memory: 512Mi
|
|
cpu: 1000m
|
|
|
|
|
|
## Monitoring Strategy
|
|
The following monitoring improvements will be implemented:
|
|
- Memory usage dashboard.
|
|
- Alert when memory exceeds 75%.
|
|
- Critical alert when memory exceeds 90%.
|
|
- Monitor container restart count.
|
|
- Track OOMKilled events.
|
|
- Synthetic health checks.
|
|
|
|
|