# Incident Postmortem Report: Memory Outage & CrashLoopBackOff

## 1. Incident Overview & Summary
- **Service Name:** Ghaymah Core Application
- **Outage Duration:** 45 Minutes
- **Severity:** High (P1 - Service Outage)
- **Status:** Resolved
- **Impact:** HTTP 502 Bad Gateway errors observed during traffic peak due to repeated pod failures.
- **Root Cause:** Container memory usage continuously exceeded its hard limit (512MB), triggering Kernel OOM Killer (`Exit Code 137`) and Kubernetes `CrashLoopBackOff`.

---

## 2. Incident Timeline
- **12:00 UTC:** Traffic surge initiated on application endpoints.
- **12:10 UTC:** Memory utilization reached 95% of defined container resource limits.
- **12:15 UTC:** Linux Kernel OOM Killer terminated the primary pod container (`OOMKilled`).
- **12:16 UTC:** Kubernetes entered a `CrashLoopBackOff` restart cycle; application became unreachable (502 Bad Gateway).
- **12:45 UTC:** SRE team identified resource constraints, updated memory limits to 2Gi, applied HPA auto-scaling, and restored normal service operations.

---

## 3. Root Cause Analysis (RCA)
1. **Inadequate Resource Allocations:** Memory limits were set statically at 512MB, which was insufficient for processing concurrent request spikes.
2. **Absence of Dynamic Scaling:** The deployment lacked auto-scaling rules, preventing horizontal pod expansion under load.

---

## 4. Remediation & Preventive Recommendations
- [x] Increased container memory resource request to 1Gi and hard limit to 2Gi.
- [x] Implemented Horizontal Pod Autoscaler (HPA) targeting memory and CPU thresholds.
- [ ] Implement proactive alerting via Prometheus Alertmanager for high memory consumption.

---

## 5. Ghaymah Auto-Scaling Policy Configuration

To prevent future memory exhaustion outages, the following Kubernetes `HorizontalPodAutoscaler` (HPA) manifest was created and applied to scale pods dynamically:

```yaml
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: ghaymah-core-app-hpa
  namespace: production
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: ghaymah-core-app
  minReplicas: 3
  maxReplicas: 10
  metrics:
  - type: Resource
    resource:
      name: memory
      target:
        type: Utilization
        averageUtilization: 75
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 80
6. Early Detection via Ghaymah Monitoring Tools
To detect and mitigate memory leakage or depletion early before service degradation occurs, we leverage Ghaymah Cloud Monitoring (Prometheus & Grafana ecosystem) as follows:

A. Key Metrics to Track:
container_memory_working_set_bytes: Measures actual memory used by the container excluding cached pages.

kube_pod_container_status_restarts_total: Tracks container restart loops triggered by OOMKilled events.

container_spec_memory_limit_bytes: Monitors usage percentage against defined resource boundaries.

B. Prometheus Alerting Rules Configuration:
We configure proactive alert rules in Prometheus/Alertmanager:

YAML
groups:
- name: ghaymah_memory_alerts
  rules:
  # 1. Early Warning Alert (Memory > 85% for 3 minutes)
  - alert: HighMemoryUsageWarning
    expr: (container_memory_working_set_bytes{container="ghaymah-core-app"} / container_spec_memory_limit_bytes{container="ghaymah-core-app"}) * 100 > 85
    for: 3m
    labels:
      severity: warning
    annotations:
      summary: "High Memory Utilization on Ghaymah Pod"
      description: "Pod {{ $labels.pod }} memory usage is above 85% for more than 3 minutes."

  # 2. Critical Alert (OOM Kill Detected)
  - alert: ContainerOOMKilledCritical
    expr: increase(kube_pod_container_status_restarts_total{container="ghaymah-core-app"}[5m]) > 0
    for: 0m
    labels:
      severity: critical
    annotations:
      summary: "Container Restarted due to OOMKilled"
      description: "Pod {{ $labels.pod }} was killed by Linux OOM Killer."
C. Grafana Visual Dashboard:
Setup a real-time Memory Threshold gauge with visual color indicators (Yellow at 75%, Red at 90%).

Enable automated PagerDuty / Slack notifications when the HighMemoryUsageWarning fires, allowing SRE engineers to intervene before pod crashes occur.