132 أسطر
4.7 KiB
Markdown
132 أسطر
4.7 KiB
Markdown
# Incident Postmortem Report
|
|
|
|
## 1. Incident Summary
|
|
|
|
**Incident Date:** YYYY-MM-DD
|
|
|
|
**Duration:** 45 minutes
|
|
|
|
**Severity:** High
|
|
|
|
**Summary:**
|
|
|
|
A production application deployed on the Ghaymah cloud platform experienced a service outage lasting 45 minutes. The application became unavailable because its container was repeatedly terminated with the **OOMKilled** status. Kubernetes continuously restarted the container, resulting in repeated failures and service disruption until the issue was identified and resolved.
|
|
## 2. Impact
|
|
|
|
- The application was unavailable for approximately 45 minutes.
|
|
- Users were unable to access the API during the outage.
|
|
- Incoming requests failed because the application containers kept restarting.
|
|
- The outage affected service availability and reduced user trust.
|
|
- No data loss was reported, but the service experienced complete downtime.
|
|
## 3. Timeline
|
|
|
|
| Time | Event |
|
|
|------|-------|
|
|
| 10:00 | Application deployed successfully. |
|
|
| 10:05 | Memory usage started increasing unexpectedly. |
|
|
| 10:08 | First container terminated with OOMKilled. |
|
|
| 10:09 | Kubernetes restarted the container automatically. |
|
|
| 10:15 | Multiple OOMKilled events occurred, causing repeated restarts. |
|
|
| 10:20 | Users reported that the application was unavailable. |
|
|
| 10:30 | Operations team investigated logs and resource usage. |
|
|
| 10:40 | Memory limits were updated and the application was restarted. |
|
|
| 10:45 | Service fully recovered and monitoring confirmed stability. |
|
|
## 4. Root Cause Analysis
|
|
|
|
The root cause of the incident was repeated **OOMKilled** events. The application consumed more memory than the configured memory limit for the container. When the memory limit was exceeded, Kubernetes terminated the container to protect the node from running out of memory.
|
|
|
|
Since the underlying memory issue was not resolved, Kubernetes automatically restarted the container. The application repeatedly exceeded the memory limit after each restart, creating a restart loop that caused the service to remain unavailable.
|
|
|
|
### Contributing Factors
|
|
|
|
- Memory limits were configured too low for the application's workload.
|
|
- No early alert was configured for high memory usage.
|
|
- The application was not tested under high memory load before deployment.
|
|
Limit = 512Mi
|
|
|
|
Application = 700Mi
|
|
|
|
↓
|
|
|
|
OOMKilled
|
|
Start
|
|
|
|
↓
|
|
|
|
OOMKilled
|
|
|
|
↓
|
|
|
|
Restart
|
|
|
|
↓
|
|
|
|
OOMKilled
|
|
|
|
↓
|
|
|
|
Restart
|
|
|
|
↓
|
|
|
|
Service Down
|
|
## 5. Resolution
|
|
|
|
The operations team investigated the application logs and Kubernetes events to identify the repeated OOMKilled status. The memory limits were increased to match the application's requirements, and the affected containers were restarted.
|
|
|
|
After applying the new configuration, the application became stable, and monitoring confirmed that no additional OOMKilled events occurred.
|
|
## 6. Recommendations
|
|
|
|
To prevent similar incidents in the future, the following actions are recommended:
|
|
|
|
- Configure appropriate CPU and memory requests and limits for all containers.
|
|
- Enable Horizontal Pod Autoscaler (HPA) to automatically increase the number of running containers during high load.
|
|
- Configure monitoring and alerting for memory usage, container restarts, and OOMKilled events.
|
|
- Perform load testing before deploying new application versions.
|
|
- Regularly review application memory consumption and optimize memory usage.
|
|
|
|
---
|
|
|
|
## 7. Auto Scaling Policy
|
|
|
|
To reduce the risk of future outages, the following auto-scaling policy is proposed:
|
|
|
|
- Minimum replicas: **2**
|
|
- Maximum replicas: **10**
|
|
- Scale out when:
|
|
- CPU usage is greater than **70%** for 5 minutes.
|
|
- Memory usage is greater than **75%** for 5 minutes.
|
|
- Scale in when CPU and memory usage remain below **40%** for 10 minutes.
|
|
- New containers must pass readiness and health checks before receiving traffic.
|
|
|
|
This policy ensures that additional containers are started before existing containers become overloaded.
|
|
|
|
---
|
|
|
|
## 8. Early Detection Using Monitoring
|
|
|
|
The issue could have been detected much earlier by using a monitoring platform.
|
|
|
|
### Monitoring Stack
|
|
|
|
- **Prometheus** collects application and Kubernetes metrics.
|
|
- **Grafana** displays dashboards for resource usage and application health.
|
|
- **Alertmanager** sends notifications by email or Slack when predefined thresholds are exceeded.
|
|
|
|
### Important Metrics
|
|
|
|
- Memory Usage
|
|
- CPU Usage
|
|
- Pod Restart Count
|
|
- Container Restart Count
|
|
- OOMKilled Events
|
|
- API Response Time
|
|
|
|
### Alert Rules
|
|
|
|
- Alert if memory usage exceeds **80%** for more than **2 minutes**.
|
|
- Alert immediately if any container enters the **OOMKilled** state.
|
|
- Alert if a pod restarts more than **3 times within 10 minutes**.
|
|
|
|
These alerts allow the operations team to investigate and resolve the issue before it causes a complete service outage.
|