105 أسطر
4.2 KiB
Markdown
105 أسطر
4.2 KiB
Markdown
# Incident Postmortem Report: Memory Outage & CrashLoopBackOff
|
|
|
|
## 1. Incident Overview & Summary
|
|
- **Service Name:** Ghaymah Core Application
|
|
- **Outage Duration:** 45 Minutes
|
|
- **Severity:** High (P1 - Service Outage)
|
|
- **Status:** Resolved
|
|
- **Impact:** HTTP 502 Bad Gateway errors observed during traffic peak due to repeated pod failures.
|
|
- **Root Cause:** Container memory usage continuously exceeded its hard limit (512MB), triggering Kernel OOM Killer (`Exit Code 137`) and Kubernetes `CrashLoopBackOff`.
|
|
|
|
---
|
|
|
|
## 2. Incident Timeline
|
|
- **12:00 UTC:** Traffic surge initiated on application endpoints.
|
|
- **12:10 UTC:** Memory utilization reached 95% of defined container resource limits.
|
|
- **12:15 UTC:** Linux Kernel OOM Killer terminated the primary pod container (`OOMKilled`).
|
|
- **12:16 UTC:** Kubernetes entered a `CrashLoopBackOff` restart cycle; application became unreachable (502 Bad Gateway).
|
|
- **12:45 UTC:** SRE team identified resource constraints, updated memory limits to 2Gi, applied HPA auto-scaling, and restored normal service operations.
|
|
|
|
---
|
|
|
|
## 3. Root Cause Analysis (RCA)
|
|
1. **Inadequate Resource Allocations:** Memory limits were set statically at 512MB, which was insufficient for processing concurrent request spikes.
|
|
2. **Absence of Dynamic Scaling:** The deployment lacked auto-scaling rules, preventing horizontal pod expansion under load.
|
|
|
|
---
|
|
|
|
## 4. Remediation & Preventive Recommendations
|
|
- [x] Increased container memory resource request to 1Gi and hard limit to 2Gi.
|
|
- [x] Implemented Horizontal Pod Autoscaler (HPA) targeting memory and CPU thresholds.
|
|
- [ ] Implement proactive alerting via Prometheus Alertmanager for high memory consumption.
|
|
|
|
---
|
|
|
|
## 5. Ghaymah Auto-Scaling Policy Configuration
|
|
|
|
To prevent future memory exhaustion outages, the following Kubernetes `HorizontalPodAutoscaler` (HPA) manifest was created and applied to scale pods dynamically:
|
|
|
|
```yaml
|
|
apiVersion: autoscaling/v2
|
|
kind: HorizontalPodAutoscaler
|
|
metadata:
|
|
name: ghaymah-core-app-hpa
|
|
namespace: production
|
|
spec:
|
|
scaleTargetRef:
|
|
apiVersion: apps/v1
|
|
kind: Deployment
|
|
name: ghaymah-core-app
|
|
minReplicas: 3
|
|
maxReplicas: 10
|
|
metrics:
|
|
- type: Resource
|
|
resource:
|
|
name: memory
|
|
target:
|
|
type: Utilization
|
|
averageUtilization: 75
|
|
- type: Resource
|
|
resource:
|
|
name: cpu
|
|
target:
|
|
type: Utilization
|
|
averageUtilization: 80
|
|
6. Early Detection via Ghaymah Monitoring Tools
|
|
To detect and mitigate memory leakage or depletion early before service degradation occurs, we leverage Ghaymah Cloud Monitoring (Prometheus & Grafana ecosystem) as follows:
|
|
|
|
A. Key Metrics to Track:
|
|
container_memory_working_set_bytes: Measures actual memory used by the container excluding cached pages.
|
|
|
|
kube_pod_container_status_restarts_total: Tracks container restart loops triggered by OOMKilled events.
|
|
|
|
container_spec_memory_limit_bytes: Monitors usage percentage against defined resource boundaries.
|
|
|
|
B. Prometheus Alerting Rules Configuration:
|
|
We configure proactive alert rules in Prometheus/Alertmanager:
|
|
|
|
YAML
|
|
groups:
|
|
- name: ghaymah_memory_alerts
|
|
rules:
|
|
# 1. Early Warning Alert (Memory > 85% for 3 minutes)
|
|
- alert: HighMemoryUsageWarning
|
|
expr: (container_memory_working_set_bytes{container="ghaymah-core-app"} / container_spec_memory_limit_bytes{container="ghaymah-core-app"}) * 100 > 85
|
|
for: 3m
|
|
labels:
|
|
severity: warning
|
|
annotations:
|
|
summary: "High Memory Utilization on Ghaymah Pod"
|
|
description: "Pod {{ $labels.pod }} memory usage is above 85% for more than 3 minutes."
|
|
|
|
# 2. Critical Alert (OOM Kill Detected)
|
|
- alert: ContainerOOMKilledCritical
|
|
expr: increase(kube_pod_container_status_restarts_total{container="ghaymah-core-app"}[5m]) > 0
|
|
for: 0m
|
|
labels:
|
|
severity: critical
|
|
annotations:
|
|
summary: "Container Restarted due to OOMKilled"
|
|
description: "Pod {{ $labels.pod }} was killed by Linux OOM Killer."
|
|
C. Grafana Visual Dashboard:
|
|
Setup a real-time Memory Threshold gauge with visual color indicators (Yellow at 75%, Red at 90%).
|
|
|
|
Enable automated PagerDuty / Slack notifications when the HighMemoryUsageWarning fires, allowing SRE engineers to intervene before pod crashes occur.
|