121 أسطر
4.6 KiB
Markdown
121 أسطر
4.6 KiB
Markdown
# Incident Postmortem & Auto-Scaling Policy — Ghaymah Cloud
|
||
|
||
**Candidate Profile**: Marwan Abdelmoneim (`marwanabdelmoneim`) | `marwantamermo@gmail.com`
|
||
**Track**: SRE — Site Reliability Engineering
|
||
**Platform**: Ghaymah Cloud (`ghaymah.systems`)
|
||
|
||
---
|
||
|
||
## 1. Executive Incident Summary
|
||
|
||
| Metric | Details |
|
||
| :--- | :--- |
|
||
| **Incident Title** | Ghaymah Production Outage due to Recurring Container `OOMKilled` Events |
|
||
| **Severity Level** | SEV-1 (Critical Outage) |
|
||
| **Outage Duration** | 45 minutes (14:15 UTC – 15:00 UTC) |
|
||
| **Impacted Services** | `api-gateway`, `user-session-service` |
|
||
| **User Impact** | ~42,000 active customer sessions disrupted (HTTP 502 Bad Gateway) |
|
||
|
||
---
|
||
|
||
## 2. Chronological Incident Timeline (UTC)
|
||
|
||
- **14:15** — Marketing promotional push causes incoming API request volume to surge from 2,500 req/s to 8,200 req/s (+228%).
|
||
- **14:17** — Memory utilization across `user-session-service` container pods hits 92% of the hard-coded 512 MiB limit.
|
||
- **14:19** — **First Pod Failure**: Linux cgroup driver triggers OOM Killer (`signal 9: SIGKILL`, Exit Code 137) on pod `user-session-service-7f8d9-x4k21`.
|
||
- **14:21** — Load Balancer redirects traffic to remaining healthy pods, accelerating memory exhaustion across surviving instances.
|
||
- **14:24** — Cascading failure: All 6 container replicas enter `OOMKilled` -> `CrashLoopBackOff` state. Error rate reaches 98.4%.
|
||
- **14:26** — SRE On-call engineer paged via PagerDuty alert: `HighErrorRate5xx > 15%`.
|
||
- **14:32** — SRE inspects container logs using Ghaymah CLI:
|
||
```bash
|
||
ghaymah container logs user-session-service --previous
|
||
# Output: Memory cgroup out of memory: Kill process 18241 (node) score 982 or sacrifice child
|
||
```
|
||
- **14:50** — Emergency Hotfix: Raised container memory limit from `512Mi` to `2Gi`, enabled garbage collector heap limit `--max-old-space-size=1536`, and deployed dual-metric HPA rule.
|
||
- **14:57** — Container pods stabilize across all Availability Zones. Error rate drops to 0.01%.
|
||
- **15:00** — Incident officially resolved.
|
||
|
||
---
|
||
|
||
## 3. Root Cause Analysis & 5-Whys
|
||
|
||
### Root Cause
|
||
Unbounded process heap growth under traffic surge combined with an improper cgroup memory limit (`512Mi`) set equal to memory request, lacking memory-based auto-scaling.
|
||
|
||
### 5-Whys Analysis
|
||
1. **Why did the application fail?** -> Containers were forcibly killed by kernel (Exit Code 137).
|
||
2. **Why were containers killed?** -> Memory usage breached allocated 512 MiB cgroup ceiling.
|
||
3. **Why did memory breach limit?** -> Session payload objects accumulated in process memory heap during traffic spike.
|
||
4. **Why didn't infrastructure scale up?** -> Autoscaler was only monitoring CPU utilization (which stayed at 45%), ignoring memory saturation.
|
||
5. **Why was memory metric missing from HPA?** -> Original scaling manifest was deployed without multi-metric memory policy guidelines.
|
||
|
||
---
|
||
|
||
## 4. Ghaymah Auto-Scaling (HPA) Policy Manifest
|
||
|
||
To prevent recurrence, deploy the following Kubernetes/Ghaymah HPA policy:
|
||
|
||
```yaml
|
||
apiVersion: autoscaling/v2
|
||
kind: HorizontalPodAutoscaler
|
||
metadata:
|
||
name: ghaymah-api-autoscaler
|
||
namespace: production
|
||
spec:
|
||
scaleTargetRef:
|
||
apiVersion: apps/v1
|
||
kind: Deployment
|
||
name: ghaymah-api-deployment
|
||
minReplicas: 10
|
||
maxReplicas: 60
|
||
metrics:
|
||
- type: Resource
|
||
resource:
|
||
name: memory
|
||
target:
|
||
type: Utilization
|
||
averageUtilization: 70 # Scales up at 70% memory limit utilization
|
||
- type: Resource
|
||
resource:
|
||
name: cpu
|
||
target:
|
||
type: Utilization
|
||
averageUtilization: 75
|
||
behavior:
|
||
scaleUp:
|
||
stabilizationWindowSeconds: 0 # Immediate scale-up on spike
|
||
policies:
|
||
- type: Percent
|
||
value: 100
|
||
periodSeconds: 15
|
||
scaleDown:
|
||
stabilizationWindowSeconds: 300 # 5-min cooldown to prevent flapping
|
||
```
|
||
|
||
---
|
||
|
||
## 5. Early Detection Monitoring Runbook
|
||
|
||
Deploy the following Prometheus alerting rules on Ghaymah Cloud:
|
||
|
||
```yaml
|
||
groups:
|
||
- name: GhaymahMemoryAlerts
|
||
rules:
|
||
- alert: ContainerMemorySaturationWarning
|
||
expr: (container_memory_working_set_bytes{container!=""} / container_spec_memory_limit_bytes{container!=""}) > 0.75
|
||
for: 3m
|
||
labels:
|
||
severity: warning
|
||
annotations:
|
||
summary: "Container {{ $labels.container }} memory > 75%"
|
||
|
||
- alert: ContainerMemoryLeakDetected
|
||
expr: deriv(container_memory_working_set_bytes{container!=""}[15m]) > 100000
|
||
for: 30m
|
||
labels:
|
||
severity: warning
|
||
annotations:
|
||
summary: "Monotonic memory leak detected in container {{ $labels.container }}"
|
||
```
|