ghaymah-exam-noran-sre/q2-postmortem/postmortem-report.md

# Incident Postmortem Report

## Application Information

| Item | Value |
|------|-------|
| **Application** | API Service on Ghaymah Platform |
| **Incident Date** | 2026-07-27 |
| **Incident Duration** | 45 Minutes (10:15 – 11:00) |
| **Severity** | High |
| **Impact** | Complete service outage. Users were unable to access the API during the incident. |

---

# 1. Executive Summary

On **2026-07-27**, the API service experienced a **45-minute outage** due to repeated **OOMKilled** events.

The application gradually consumed more memory because of a memory leak while processing large requests. Once the container exceeded its allocated memory limit, the platform terminated and restarted it automatically. This restart cycle repeated continuously, resulting in complete service unavailability.

The immediate mitigation was increasing the container memory allocation and restarting the service. Long-term corrective actions include fixing the memory leak, improving monitoring, and implementing an auto-scaling policy based on resource utilization.

---

# 2. Incident Timeline

| Time | Event |
|------|-------|
| **10:15** | Memory usage started increasing on one of the application containers. |
| **10:22** | First container was terminated with **OOMKilled** and automatically restarted by the platform. |
| **10:25 – 10:45** | Multiple containers repeatedly entered restart loops, reducing the application's availability. |
| **10:45** | Monitoring system generated alerts indicating complete service outage. |
| **10:50** | SRE team began investigating the incident and identified abnormal memory usage. |
| **11:00** | Container memory limit increased from **512 MiB** to **1 GiB** and services were restarted successfully. |
| **11:15** | Additional monitoring and temporary mitigation measures were applied. |

---

# 3. Root Cause Analysis

## Immediate Cause

The application exhausted the available memory, causing the platform to terminate the container with an **OOMKilled** event.

## Root Cause

A memory leak inside the request processing logic (or an external library) continuously retained memory instead of releasing it after requests were completed.

## Contributing Factors

- Sudden increase in traffic (approximately **3× normal load**).
- Insufficient memory allocation.
- No automatic scaling based on memory utilization.
- Lack of early warning alerts before reaching the memory limit.

---

# 4. Recommendations

## Short-Term Actions

- Increase container memory limits.
- Monitor memory consumption continuously.
- Restart unhealthy containers automatically.

## Long-Term Improvements

- Fix the memory leak in the application.
- Configure appropriate container memory **requests** and **limits**.
- Enable memory-based auto-scaling.
- Configure proactive alerting.
- Perform regular load and stress testing.
- Maintain an incident response runbook for OOMKilled events.

---

# 5. Auto-Scaling Policy

To prevent similar incidents, the following auto-scaling policy is recommended.

| Configuration | Value |
|--------------|-------|
| **Metric** | Average Container Memory Usage |
| **Scale-Out Threshold** | Memory usage > **70%** for **2 minutes** |
| **Scale-Out Action** | Add one container instance |
| **Minimum Replicas** | 2 |
| **Maximum Replicas** | 10 |
| **Cooldown Period** | 5 Minutes |
| **Scale-In Threshold** | Memory usage < **40%** for **10 minutes** |
| **Scale-In Action** | Remove one container instance |

### Example Auto-Scaling Configuration

```yaml
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: api-hpa
spec:
  minReplicas: 2
  maxReplicas: 10
  metrics:
    - type: Resource
      resource:
        name: memory
        target:
          type: Utilization
          averageUtilization: 70
```

> **Note:** The YAML above is an example of a memory-based auto-scaling policy and illustrates the desired behavior conceptually.

---

# 6. Early Detection Using Ghaymah Monitoring

The following monitoring capabilities should be configured to detect similar incidents before they impact users.

## Resource Monitoring

Track real-time metrics including:

- Container Memory Usage
- CPU Utilization
- Container Restart Count
- Container Health Status

---

## Alerting

Configure alerts such as:

| Condition | Action |
|-----------|--------|
| Memory usage > 60% for 3 minutes | Send Email / Slack / Webhook notification |
| Multiple container restarts | Trigger High-Priority Alert |
| Service unavailable | Immediate Critical Alert |

---

## Event Monitoring

Monitor platform events including:

- OOMKilled
- Container Restart
- CrashLoop
- Failed Deployments

---

## Dashboard

Create a monitoring dashboard displaying:

- Current Container Status
- Memory Utilization
- CPU Utilization
- Response Time
- Request Count
- Restart Count
- OOMKilled Events
- Service Availability (Uptime)

---

## Trend Analysis

Analyze historical metrics to identify:

- Gradual memory growth
- Traffic spikes
- Abnormal restart frequency
- Resource utilization trends

This enables proactive detection before a service outage occurs.

---

# 7. Lessons Learned

The incident highlighted several important operational improvements:

- Proper memory sizing is essential.
- Continuous monitoring should be proactive rather than reactive.
- Memory-based auto-scaling reduces service interruption.
- Early alerts significantly reduce recovery time.
- Load testing helps identify memory leaks before production deployment.

---

# Action Items

| Priority | Action | Status |
|----------|--------|--------|
| High | Fix memory leak | Planned |
| High | Configure memory alerts | Planned |
| High | Enable auto-scaling | Planned |
| Medium | Perform load testing | Planned |
| Medium | Update operational runbook | Planned |

---

# Report Information

| Item | Value |
|------|-------|
| **Prepared By** | SRE Team |
| **Platform** | Ghaymah Systems |
| **Report Date** | 2026-07-27 |
| **Incident Type** | Repeated OOMKilled |
| **Status** | Closed |