Incident Postmortem & Auto-Scaling Policy — Ghaymah Cloud

Candidate Profile: Marwan Abdelmoneim (marwanabdelmoneim) | marwantamermo@gmail.com
Track: SRE — Site Reliability Engineering
Platform: Ghaymah Cloud (ghaymah.systems)

1. Executive Incident Summary

Metric	Details
Incident Title	Ghaymah Production Outage due to Recurring Container `OOMKilled` Events
Severity Level	SEV-1 (Critical Outage)
Outage Duration	45 minutes (14:15 UTC – 15:00 UTC)
Impacted Services	`api-gateway`, `user-session-service`
User Impact	~42,000 active customer sessions disrupted (HTTP 502 Bad Gateway)

2. Chronological Incident Timeline (UTC)

14:15 — Marketing promotional push causes incoming API request volume to surge from 2,500 req/s to 8,200 req/s (+228%).
14:17 — Memory utilization across user-session-service container pods hits 92% of the hard-coded 512 MiB limit.
14:19 — First Pod Failure: Linux cgroup driver triggers OOM Killer (signal 9: SIGKILL, Exit Code 137) on pod user-session-service-7f8d9-x4k21.
14:21 — Load Balancer redirects traffic to remaining healthy pods, accelerating memory exhaustion across surviving instances.
14:24 — Cascading failure: All 6 container replicas enter OOMKilled -> CrashLoopBackOff state. Error rate reaches 98.4%.
14:26 — SRE On-call engineer paged via PagerDuty alert: HighErrorRate5xx > 15%.

14:32 — SRE inspects container logs using Ghaymah CLI:

ghaymah container logs user-session-service --previous
# Output: Memory cgroup out of memory: Kill process 18241 (node) score 982 or sacrifice child

14:50 — Emergency Hotfix: Raised container memory limit from 512Mi to 2Gi, enabled garbage collector heap limit --max-old-space-size=1536, and deployed dual-metric HPA rule.
14:57 — Container pods stabilize across all Availability Zones. Error rate drops to 0.01%.
15:00 — Incident officially resolved.

3. Root Cause Analysis & 5-Whys

Root Cause

Unbounded process heap growth under traffic surge combined with an improper cgroup memory limit (512Mi) set equal to memory request, lacking memory-based auto-scaling.

5-Whys Analysis

Why did the application fail? -> Containers were forcibly killed by kernel (Exit Code 137).
Why were containers killed? -> Memory usage breached allocated 512 MiB cgroup ceiling.
Why did memory breach limit? -> Session payload objects accumulated in process memory heap during traffic spike.
Why didn't infrastructure scale up? -> Autoscaler was only monitoring CPU utilization (which stayed at 45%), ignoring memory saturation.
Why was memory metric missing from HPA? -> Original scaling manifest was deployed without multi-metric memory policy guidelines.

4. Ghaymah Auto-Scaling (HPA) Policy Manifest

To prevent recurrence, deploy the following Kubernetes/Ghaymah HPA policy:

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: ghaymah-api-autoscaler
  namespace: production
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: ghaymah-api-deployment
  minReplicas: 10
  maxReplicas: 60
  metrics:
    - type: Resource
      resource:
        name: memory
        target:
          type: Utilization
          averageUtilization: 70  # Scales up at 70% memory limit utilization
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 75
  behavior:
    scaleUp:
      stabilizationWindowSeconds: 0 # Immediate scale-up on spike
      policies:
        - type: Percent
          value: 100
          periodSeconds: 15
    scaleDown:
      stabilizationWindowSeconds: 300 # 5-min cooldown to prevent flapping

5. Early Detection Monitoring Runbook

Deploy the following Prometheus alerting rules on Ghaymah Cloud:

groups:
  - name: GhaymahMemoryAlerts
    rules:
      - alert: ContainerMemorySaturationWarning
        expr: (container_memory_working_set_bytes{container!=""} / container_spec_memory_limit_bytes{container!=""}) > 0.75
        for: 3m
        labels:
          severity: warning
        annotations:
          summary: "Container {{ $labels.container }} memory > 75%"
      
      - alert: ContainerMemoryLeakDetected
        expr: deriv(container_memory_working_set_bytes{container!=""}[15m]) > 100000
        for: 30m
        labels:
          severity: warning
        annotations:
          summary: "Monotonic memory leak detected in container {{ $labels.container }}"

4.6 KiB خام الرابط الدائم اللوم التاريخ إلغاء الهروب الهروب