feat: complete SRE exam deliverables for Ghaymah & Qabilah platform submission
هذا الالتزام موجود في:
120
q2-postmortem/postmortem-report.md
Normal file
120
q2-postmortem/postmortem-report.md
Normal file
@@ -0,0 +1,120 @@
|
||||
# Incident Postmortem & Auto-Scaling Policy — Ghaymah Cloud
|
||||
|
||||
**Candidate Profile**: Marwan Abdelmoneim (`marwanabdelmoneim`) | `marwantamermo@gmail.com`
|
||||
**Track**: SRE — Site Reliability Engineering
|
||||
**Platform**: Ghaymah Cloud (`ghaymah.systems`)
|
||||
|
||||
---
|
||||
|
||||
## 1. Executive Incident Summary
|
||||
|
||||
| Metric | Details |
|
||||
| :--- | :--- |
|
||||
| **Incident Title** | Ghaymah Production Outage due to Recurring Container `OOMKilled` Events |
|
||||
| **Severity Level** | SEV-1 (Critical Outage) |
|
||||
| **Outage Duration** | 45 minutes (14:15 UTC – 15:00 UTC) |
|
||||
| **Impacted Services** | `api-gateway`, `user-session-service` |
|
||||
| **User Impact** | ~42,000 active customer sessions disrupted (HTTP 502 Bad Gateway) |
|
||||
|
||||
---
|
||||
|
||||
## 2. Chronological Incident Timeline (UTC)
|
||||
|
||||
- **14:15** — Marketing promotional push causes incoming API request volume to surge from 2,500 req/s to 8,200 req/s (+228%).
|
||||
- **14:17** — Memory utilization across `user-session-service` container pods hits 92% of the hard-coded 512 MiB limit.
|
||||
- **14:19** — **First Pod Failure**: Linux cgroup driver triggers OOM Killer (`signal 9: SIGKILL`, Exit Code 137) on pod `user-session-service-7f8d9-x4k21`.
|
||||
- **14:21** — Load Balancer redirects traffic to remaining healthy pods, accelerating memory exhaustion across surviving instances.
|
||||
- **14:24** — Cascading failure: All 6 container replicas enter `OOMKilled` -> `CrashLoopBackOff` state. Error rate reaches 98.4%.
|
||||
- **14:26** — SRE On-call engineer paged via PagerDuty alert: `HighErrorRate5xx > 15%`.
|
||||
- **14:32** — SRE inspects container logs using Ghaymah CLI:
|
||||
```bash
|
||||
ghaymah container logs user-session-service --previous
|
||||
# Output: Memory cgroup out of memory: Kill process 18241 (node) score 982 or sacrifice child
|
||||
```
|
||||
- **14:50** — Emergency Hotfix: Raised container memory limit from `512Mi` to `2Gi`, enabled garbage collector heap limit `--max-old-space-size=1536`, and deployed dual-metric HPA rule.
|
||||
- **14:57** — Container pods stabilize across all Availability Zones. Error rate drops to 0.01%.
|
||||
- **15:00** — Incident officially resolved.
|
||||
|
||||
---
|
||||
|
||||
## 3. Root Cause Analysis & 5-Whys
|
||||
|
||||
### Root Cause
|
||||
Unbounded process heap growth under traffic surge combined with an improper cgroup memory limit (`512Mi`) set equal to memory request, lacking memory-based auto-scaling.
|
||||
|
||||
### 5-Whys Analysis
|
||||
1. **Why did the application fail?** -> Containers were forcibly killed by kernel (Exit Code 137).
|
||||
2. **Why were containers killed?** -> Memory usage breached allocated 512 MiB cgroup ceiling.
|
||||
3. **Why did memory breach limit?** -> Session payload objects accumulated in process memory heap during traffic spike.
|
||||
4. **Why didn't infrastructure scale up?** -> Autoscaler was only monitoring CPU utilization (which stayed at 45%), ignoring memory saturation.
|
||||
5. **Why was memory metric missing from HPA?** -> Original scaling manifest was deployed without multi-metric memory policy guidelines.
|
||||
|
||||
---
|
||||
|
||||
## 4. Ghaymah Auto-Scaling (HPA) Policy Manifest
|
||||
|
||||
To prevent recurrence, deploy the following Kubernetes/Ghaymah HPA policy:
|
||||
|
||||
```yaml
|
||||
apiVersion: autoscaling/v2
|
||||
kind: HorizontalPodAutoscaler
|
||||
metadata:
|
||||
name: ghaymah-api-autoscaler
|
||||
namespace: production
|
||||
spec:
|
||||
scaleTargetRef:
|
||||
apiVersion: apps/v1
|
||||
kind: Deployment
|
||||
name: ghaymah-api-deployment
|
||||
minReplicas: 10
|
||||
maxReplicas: 60
|
||||
metrics:
|
||||
- type: Resource
|
||||
resource:
|
||||
name: memory
|
||||
target:
|
||||
type: Utilization
|
||||
averageUtilization: 70 # Scales up at 70% memory limit utilization
|
||||
- type: Resource
|
||||
resource:
|
||||
name: cpu
|
||||
target:
|
||||
type: Utilization
|
||||
averageUtilization: 75
|
||||
behavior:
|
||||
scaleUp:
|
||||
stabilizationWindowSeconds: 0 # Immediate scale-up on spike
|
||||
policies:
|
||||
- type: Percent
|
||||
value: 100
|
||||
periodSeconds: 15
|
||||
scaleDown:
|
||||
stabilizationWindowSeconds: 300 # 5-min cooldown to prevent flapping
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 5. Early Detection Monitoring Runbook
|
||||
|
||||
Deploy the following Prometheus alerting rules on Ghaymah Cloud:
|
||||
|
||||
```yaml
|
||||
groups:
|
||||
- name: GhaymahMemoryAlerts
|
||||
rules:
|
||||
- alert: ContainerMemorySaturationWarning
|
||||
expr: (container_memory_working_set_bytes{container!=""} / container_spec_memory_limit_bytes{container!=""}) > 0.75
|
||||
for: 3m
|
||||
labels:
|
||||
severity: warning
|
||||
annotations:
|
||||
summary: "Container {{ $labels.container }} memory > 75%"
|
||||
|
||||
- alert: ContainerMemoryLeakDetected
|
||||
expr: deriv(container_memory_working_set_bytes{container!=""}[15m]) > 100000
|
||||
for: 30m
|
||||
labels:
|
||||
severity: warning
|
||||
annotations:
|
||||
summary: "Monotonic memory leak detected in container {{ $labels.container }}"
|
||||
```
|
||||
المرجع في مشكلة جديدة
حظر مستخدم