121 أسطر
3.1 KiB
Markdown
121 أسطر
3.1 KiB
Markdown
# Postmortem Report
|
|
|
|
## Incident Summary
|
|
|
|
**Incident Title:** Repeated OOMKilled Causing Application Downtime
|
|
|
|
**Date:** 2026-07-28
|
|
|
|
**Duration:** 45 minutes
|
|
|
|
**Severity:** High (SEV-1)
|
|
|
|
**Impact:**
|
|
The application became unavailable for 45 minutes because the container was repeatedly terminated with the `OOMKilled` status. During the outage, users were unable to access the application, resulting in service disruption.
|
|
|
|
---
|
|
|
|
# Timeline
|
|
|
|
| Time | Event |
|
|
|------|-------|
|
|
| 14:00 | Increased traffic caused memory usage to rise. |
|
|
| 14:05 | Container exceeded its memory limit and was terminated (OOMKilled). |
|
|
| 14:06 | Kubernetes restarted the container automatically. |
|
|
| 14:10 - 14:40 | Continuous restart loop occurred due to repeated OOMKilled events. |
|
|
| 14:25 | Monitoring system generated high memory usage alerts. |
|
|
| 14:35 | SRE team began investigating pod events and resource metrics. |
|
|
| 14:45 | Memory limit was increased and the deployment was restarted. |
|
|
| 14:45 | Application recovered successfully. |
|
|
|
|
---
|
|
|
|
# Root Cause Analysis
|
|
|
|
### Primary Root Cause
|
|
|
|
The application exceeded the configured memory limit. When the container reached the limit, the Linux Out Of Memory (OOM) Killer terminated the process.
|
|
|
|
### Contributing Factors
|
|
|
|
- Memory limits were configured too low.
|
|
- No Horizontal Pod Autoscaler (HPA) was configured.
|
|
- Monitoring alerts were triggered after the application became unstable.
|
|
- No load testing had been performed before production deployment.
|
|
|
|
---
|
|
|
|
# Resolution
|
|
|
|
The following actions restored the service:
|
|
|
|
- Increased container memory limits.
|
|
- Restarted the deployment.
|
|
- Verified application health using `/health`.
|
|
- Confirmed stable memory usage.
|
|
|
|
---
|
|
|
|
# Preventive Actions
|
|
|
|
### Immediate
|
|
|
|
- Increase memory requests and limits.
|
|
- Add memory usage alerts.
|
|
- Perform load testing before deployment.
|
|
|
|
### Long-term
|
|
|
|
- Configure Horizontal Pod Autoscaler (HPA).
|
|
- Implement automatic scaling based on CPU and Memory.
|
|
- Enable centralized logging.
|
|
- Create SLOs and alerting rules.
|
|
- Review resource requests for every deployment.
|
|
|
|
---
|
|
|
|
# Auto-Scaling Policy
|
|
|
|
To prevent similar incidents:
|
|
|
|
- Minimum Replicas: **2**
|
|
- Maximum Replicas: **10**
|
|
- Target CPU Utilization: **70%**
|
|
- Target Memory Utilization: **75%**
|
|
- Scale Up immediately when thresholds are exceeded.
|
|
- Scale Down gradually after 5 minutes of stable utilization.
|
|
- Configure PodDisruptionBudget to maintain availability.
|
|
|
|
---
|
|
|
|
# Early Detection Using Ghaymah Monitoring
|
|
|
|
The issue can be detected early by monitoring:
|
|
|
|
- Container Memory Usage
|
|
- Pod Restart Count
|
|
- OOMKilled Events
|
|
- Container Health Checks
|
|
- Response Time
|
|
- Error Rate (5xx)
|
|
- Request Rate
|
|
- Application Availability
|
|
|
|
Recommended alerts:
|
|
|
|
- Memory Usage > 80%
|
|
- Restart Count > 3 within 5 minutes
|
|
- Health Check Failure
|
|
- Response Time > 500 ms
|
|
- Availability below 99%
|
|
|
|
---
|
|
|
|
# Lessons Learned
|
|
|
|
- Configure appropriate resource limits.
|
|
- Enable proactive monitoring and alerting.
|
|
- Test the application under expected production load.
|
|
- Implement autoscaling before deploying production workloads.
|
|
- Continuously review resource utilization trends.
|