3.1 KiB
Incident Postmortem Report
Application Outage Due to Repeated OOMKilled Events
1. Incident Overview
Incident ID: INC-2026-001 Severity: SEV-2 Duration: 45 minutes Service: Ghaymah Hosted Application Status: Resolved
The application experienced repeated downtime caused by Kubernetes OOMKilled events. The container exceeded its allocated memory limit, causing Kubernetes to terminate and restart the container repeatedly.
2. Executive Summary
On July 28, 2026, the application experienced intermittent availability issues for approximately 45 minutes.
The root cause was excessive memory consumption inside the application container. Once the container exceeded its configured memory limit, Kubernetes terminated the container with an OOMKilled event.
The service recovered after adjusting resource configuration and implementing a better scaling strategy.
3. Impact
During the incident:
- Users experienced 502/503 errors.
- API requests failed intermittently.
- Application availability decreased for 45 minutes.
- No data loss occurred because the application was stateless.
4. Detection
The incident was detected through:
- Kubernetes container restart events.
- OOMKilled status reported by the platform.
- Increased application response latency.
- Monitoring alerts triggered by high memory utilization.
5. Timeline
| Time | Event |
|---|---|
| 10:00 | Traffic increased above normal level |
| 10:02 | Container memory usage exceeded 90% |
| 10:04 | Kubernetes terminated container (OOMKilled) |
| 10:05 | New container instance started |
| 10:10 | Memory increased again and container crashed |
| 10:45 | Service stabilized after mitigation |
6. Root Cause Analysis
Immediate Cause
The container exceeded its configured memory limit.
Root Cause
The application was running with insufficient memory resources and no horizontal scaling mechanism. Possible contributing factors:
- Memory leak inside the application.
- Missing memory-based autoscaling.
- No early warning alerts.
- Single container handling all traffic.
7. Resolution
The incident was resolved by:
- Increasing container memory allocation.
- Restarting unhealthy workloads.
- Reviewing application memory usage.
- Preparing an autoscaling policy.
8. Preventive Actions
| Action | Priority |
|---|---|
| Enable Horizontal Pod Autoscaler | High |
| Add memory utilization alerts | High |
| Perform memory profiling | Medium |
| Improve health checks | Medium |
| Add resource limits based on metrics | High |
autoscaling: minReplicas: 2 maxReplicas: 8
metrics: memory: targetUtilization: 70
cpu:
targetUtilization: 75
behavior: scaleUp: stabilizationWindowSeconds: 60
scaleDown:
stabilizationWindowSeconds: 300
resources: requests: memory: 256Mi cpu: 500m
limits: memory: 512Mi cpu: 1000m
Monitoring Strategy
The following monitoring improvements will be implemented:
- Memory usage dashboard.
- Alert when memory exceeds 75%.
- Critical alert when memory exceeds 90%.
- Monitor container restart count.
- Track OOMKilled events.
- Synthetic health checks.