2.5 KiB
Postmortem Report
Incident Summary
On July 28, 2026, the application experienced repeated OOMKilled events, causing service interruption for approximately 45 minutes. During this period, users were unable to access the application.
Timeline
- 10:00 AM - Application deployed successfully.
- 10:12 AM - Memory usage started increasing.
- 10:18 AM - First OOMKilled event occurred.
- 10:20 AM - Kubernetes restarted the container.
- 10:24 AM - Multiple OOMKilled events repeated.
- 10:45 AM - Monitoring system alerted the operations team.
- 11:00 AM - Memory limit increased and memory leak fixed.
- 11:03 AM - Service fully restored.
Root Cause
The application contained a memory leak that caused continuous memory consumption. The container exceeded its memory limit, resulting in repeated OOMKilled events and continuous restarts.
Recommendations
- Set appropriate memory requests and limits.
- Enable Horizontal Pod Autoscaler (HPA).
- Monitor memory usage continuously.
- Configure alerting before memory reaches critical levels.
- Perform load testing before production deployment.
Auto Scaling Policy
Objective
Prevent service interruption caused by high memory usage and repeated OOMKilled events.
Scaling Policy
- Minimum Replicas: 2
- Maximum Replicas: 10
Scale Out
Increase the number of containers when:
- CPU usage > 70% for 2 minutes.
- Memory usage > 75% for 2 minutes.
Increase by 2 replicas at a time.
Scale In
Decrease the number of containers when:
- CPU usage < 40% for 10 minutes.
- Memory usage < 50% for 10 minutes.
Remove only one replica at a time.
Additional Protection
- Configure memory requests and limits.
- Enable readiness and liveness probes.
- Prevent scheduling on unhealthy nodes.
Early Detection Using Cloud Monitoring
The issue can be detected early using cloud monitoring tools by continuously monitoring application and infrastructure metrics.
Metrics to Monitor
- Memory Usage
- CPU Usage
- Container Restarts
- OOMKilled Events
- Pod Status
- Application Response Time
Alerts
Configure alerts when:
- Memory usage exceeds 80%.
- Container restarts more than 3 times within 5 minutes.
- OOMKilled event is detected.
- Response time exceeds 500 ms.
Monitoring Tools
- Prometheus
- Grafana
- Kubernetes Metrics Server
- Cloud Monitoring Platform
These tools provide dashboards, real-time metrics, and alert notifications to help engineers respond before the service becomes unavailable.