# Incident Postmortem & Auto-Scaling Policy — Ghaymah Cloud

**Candidate Profile**: Marwan Abdelmoneim (`marwanabdelmoneim`) | `marwantamermo@gmail.com`  
**Track**: SRE — Site Reliability Engineering  
**Platform**: Ghaymah Cloud (`ghaymah.systems`)  

---

## 1. Executive Incident Summary

| Metric | Details |
| :--- | :--- |
| **Incident Title** | Ghaymah Production Outage due to Recurring Container `OOMKilled` Events |
| **Severity Level** | SEV-1 (Critical Outage) |
| **Outage Duration** | 45 minutes (14:15 UTC – 15:00 UTC) |
| **Impacted Services** | `api-gateway`, `user-session-service` |
| **User Impact** | ~42,000 active customer sessions disrupted (HTTP 502 Bad Gateway) |

---

## 2. Chronological Incident Timeline (UTC)

- **14:15** — Marketing promotional push causes incoming API request volume to surge from 2,500 req/s to 8,200 req/s (+228%).
- **14:17** — Memory utilization across `user-session-service` container pods hits 92% of the hard-coded 512 MiB limit.
- **14:19** — **First Pod Failure**: Linux cgroup driver triggers OOM Killer (`signal 9: SIGKILL`, Exit Code 137) on pod `user-session-service-7f8d9-x4k21`.
- **14:21** — Load Balancer redirects traffic to remaining healthy pods, accelerating memory exhaustion across surviving instances.
- **14:24** — Cascading failure: All 6 container replicas enter `OOMKilled` -> `CrashLoopBackOff` state. Error rate reaches 98.4%.
- **14:26** — SRE On-call engineer paged via PagerDuty alert: `HighErrorRate5xx > 15%`.
- **14:32** — SRE inspects container logs using Ghaymah CLI:
  ```bash
  ghaymah container logs user-session-service --previous
  # Output: Memory cgroup out of memory: Kill process 18241 (node) score 982 or sacrifice child
  ```
- **14:50** — Emergency Hotfix: Raised container memory limit from `512Mi` to `2Gi`, enabled garbage collector heap limit `--max-old-space-size=1536`, and deployed dual-metric HPA rule.
- **14:57** — Container pods stabilize across all Availability Zones. Error rate drops to 0.01%.
- **15:00** — Incident officially resolved.

---

## 3. Root Cause Analysis & 5-Whys

### Root Cause
Unbounded process heap growth under traffic surge combined with an improper cgroup memory limit (`512Mi`) set equal to memory request, lacking memory-based auto-scaling.

### 5-Whys Analysis
1. **Why did the application fail?** -> Containers were forcibly killed by kernel (Exit Code 137).
2. **Why were containers killed?** -> Memory usage breached allocated 512 MiB cgroup ceiling.
3. **Why did memory breach limit?** -> Session payload objects accumulated in process memory heap during traffic spike.
4. **Why didn't infrastructure scale up?** -> Autoscaler was only monitoring CPU utilization (which stayed at 45%), ignoring memory saturation.
5. **Why was memory metric missing from HPA?** -> Original scaling manifest was deployed without multi-metric memory policy guidelines.

---

## 4. Ghaymah Auto-Scaling (HPA) Policy Manifest

To prevent recurrence, deploy the following Kubernetes/Ghaymah HPA policy:

```yaml
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: ghaymah-api-autoscaler
  namespace: production
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: ghaymah-api-deployment
  minReplicas: 10
  maxReplicas: 60
  metrics:
    - type: Resource
      resource:
        name: memory
        target:
          type: Utilization
          averageUtilization: 70  # Scales up at 70% memory limit utilization
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 75
  behavior:
    scaleUp:
      stabilizationWindowSeconds: 0 # Immediate scale-up on spike
      policies:
        - type: Percent
          value: 100
          periodSeconds: 15
    scaleDown:
      stabilizationWindowSeconds: 300 # 5-min cooldown to prevent flapping
```

---

## 5. Early Detection Monitoring Runbook

Deploy the following Prometheus alerting rules on Ghaymah Cloud:

```yaml
groups:
  - name: GhaymahMemoryAlerts
    rules:
      - alert: ContainerMemorySaturationWarning
        expr: (container_memory_working_set_bytes{container!=""} / container_spec_memory_limit_bytes{container!=""}) > 0.75
        for: 3m
        labels:
          severity: warning
        annotations:
          summary: "Container {{ $labels.container }} memory > 75%"
      
      - alert: ContainerMemoryLeakDetected
        expr: deriv(container_memory_working_set_bytes{container!=""}[15m]) > 100000
        for: 30m
        labels:
          severity: warning
        annotations:
          summary: "Monotonic memory leak detected in container {{ $labels.container }}"
```