4.7 KiB
Incident Postmortem Report
1. Incident Summary
Incident Date: YYYY-MM-DD
Duration: 45 minutes
Severity: High
Summary:
A production application deployed on the Ghaymah cloud platform experienced a service outage lasting 45 minutes. The application became unavailable because its container was repeatedly terminated with the OOMKilled status. Kubernetes continuously restarted the container, resulting in repeated failures and service disruption until the issue was identified and resolved.
2. Impact
- The application was unavailable for approximately 45 minutes.
- Users were unable to access the API during the outage.
- Incoming requests failed because the application containers kept restarting.
- The outage affected service availability and reduced user trust.
- No data loss was reported, but the service experienced complete downtime.
3. Timeline
| Time | Event |
|---|---|
| 10:00 | Application deployed successfully. |
| 10:05 | Memory usage started increasing unexpectedly. |
| 10:08 | First container terminated with OOMKilled. |
| 10:09 | Kubernetes restarted the container automatically. |
| 10:15 | Multiple OOMKilled events occurred, causing repeated restarts. |
| 10:20 | Users reported that the application was unavailable. |
| 10:30 | Operations team investigated logs and resource usage. |
| 10:40 | Memory limits were updated and the application was restarted. |
| 10:45 | Service fully recovered and monitoring confirmed stability. |
4. Root Cause Analysis
The root cause of the incident was repeated OOMKilled events. The application consumed more memory than the configured memory limit for the container. When the memory limit was exceeded, Kubernetes terminated the container to protect the node from running out of memory.
Since the underlying memory issue was not resolved, Kubernetes automatically restarted the container. The application repeatedly exceeded the memory limit after each restart, creating a restart loop that caused the service to remain unavailable.
Contributing Factors
- Memory limits were configured too low for the application's workload.
- No early alert was configured for high memory usage.
- The application was not tested under high memory load before deployment. Limit = 512Mi
Application = 700Mi
↓
OOMKilled Start
↓
OOMKilled
↓
Restart
↓
OOMKilled
↓
Restart
↓
Service Down
5. Resolution
The operations team investigated the application logs and Kubernetes events to identify the repeated OOMKilled status. The memory limits were increased to match the application's requirements, and the affected containers were restarted.
After applying the new configuration, the application became stable, and monitoring confirmed that no additional OOMKilled events occurred.
6. Recommendations
To prevent similar incidents in the future, the following actions are recommended:
- Configure appropriate CPU and memory requests and limits for all containers.
- Enable Horizontal Pod Autoscaler (HPA) to automatically increase the number of running containers during high load.
- Configure monitoring and alerting for memory usage, container restarts, and OOMKilled events.
- Perform load testing before deploying new application versions.
- Regularly review application memory consumption and optimize memory usage.
7. Auto Scaling Policy
To reduce the risk of future outages, the following auto-scaling policy is proposed:
- Minimum replicas: 2
- Maximum replicas: 10
- Scale out when:
- CPU usage is greater than 70% for 5 minutes.
- Memory usage is greater than 75% for 5 minutes.
- Scale in when CPU and memory usage remain below 40% for 10 minutes.
- New containers must pass readiness and health checks before receiving traffic.
This policy ensures that additional containers are started before existing containers become overloaded.
8. Early Detection Using Monitoring
The issue could have been detected much earlier by using a monitoring platform.
Monitoring Stack
- Prometheus collects application and Kubernetes metrics.
- Grafana displays dashboards for resource usage and application health.
- Alertmanager sends notifications by email or Slack when predefined thresholds are exceeded.
Important Metrics
- Memory Usage
- CPU Usage
- Pod Restart Count
- Container Restart Count
- OOMKilled Events
- API Response Time
Alert Rules
- Alert if memory usage exceeds 80% for more than 2 minutes.
- Alert immediately if any container enters the OOMKilled state.
- Alert if a pod restarts more than 3 times within 10 minutes.
These alerts allow the operations team to investigate and resolve the issue before it causes a complete service outage.