3.7 KiB
Postmortem Report – Repeated OOMKilled Incident
Incident Date: YYYY-MM-DD
Duration: 45 minutes
Service: Application hosted on Ghaymah Systems
Severity: High
1. Summary
The application experienced repeated OOMKilled events, causing continuous container restarts and making the service unavailable for approximately 45 minutes.
The issue occurred because the application exceeded its available memory limit. Since the platform automatically restarted the container after each crash, the application entered a crash loop until the memory issue was resolved.
2. Timeline
| Time | Event |
|---|---|
| 10:00 | New application version deployed |
| 10:05 | Memory usage started increasing rapidly |
| 10:10 | First OOMKilled event occurred |
| 10:11 | Platform restarted the container |
| 10:14 | Container exceeded memory limit again |
| 10:15 | Second OOMKilled event |
| 10:20 | Multiple restart attempts continued |
| 10:35 | Engineering team identified abnormal memory consumption |
| 10:45 | Memory issue resolved and application recovered |
3. Root Cause Analysis
Immediate Cause
The application consumed more memory than the container's configured memory limit, causing the Linux Out-Of-Memory (OOM) Killer to terminate the process.
Root Cause
Possible contributing factors include:
- Memory leak in the application
- Large objects remaining in memory
- Insufficient memory limits for production workload
- Lack of early monitoring and alerting
4. Impact
- Service unavailable for 45 minutes
- Users could not access the application
- Multiple container restarts
- Increased error rate and failed requests
5. Recommendations
Short-Term
- Increase container memory limit.
- Restart affected containers.
- Verify application memory usage after deployment.
- Roll back if abnormal memory growth is detected.
Long-Term
- Fix memory leaks.
- Perform load testing before production deployments.
- Configure monitoring and alerting for memory usage.
- Monitor container restart count.
- Enable automatic scaling.
- Establish deployment health checks.
Auto-Scaling Policy
Objective
Prevent service outages caused by high resource utilization.
Scale-Out Rules
- Add one new instance when:
- CPU usage > 70% for 5 minutes.
- Memory usage > 80% for 5 minutes.
- Average response time > 500 ms.
Scale-In Rules
- Remove one instance when:
- CPU usage < 30% for 10 minutes.
- Memory usage < 40% for 10 minutes.
Minimum Instances
- 2 running instances
Maximum Instances
- 10 running instances
Health Checks
- Check
/healthevery 30 seconds. - Replace unhealthy containers automatically.
Early Detection Using Monitoring
To detect similar issues before they cause downtime, monitor the following metrics:
Infrastructure Metrics
- Memory Usage
- Memory Limit
- CPU Usage
- Container Restarts
- OOMKilled Events
- Disk Usage
Application Metrics
- HTTP Response Time
- Request Rate
- Error Rate (4xx / 5xx)
- Active Connections
Alerts
Create alerts when:
- Memory usage exceeds 80%
- Container restart count increases
- OOMKilled event detected
- Response time exceeds 500 ms
- Error rate exceeds 5%
Monitoring Stack
Example monitoring solution:
- Prometheus (metrics collection)
- Grafana (dashboards)
- Alertmanager (notifications)
- Email / Slack / Microsoft Teams notifications
Lessons Learned
- Resource limits should be validated before deployment.
- Memory consumption should be continuously monitored.
- Health checks and alerts must be configured for production services.
- Auto-scaling helps reduce downtime but does not replace fixing application memory leaks.
- Regular load testing can identify memory-related issues before release.