# Postmortem Report

## Incident Summary

**Incident Title:** Repeated OOMKilled Causing Application Downtime

**Date:** 2026-07-28

**Duration:** 45 minutes

**Severity:** High (SEV-1)

**Impact:**
The application became unavailable for 45 minutes because the container was repeatedly terminated with the `OOMKilled` status. During the outage, users were unable to access the application, resulting in service disruption.

---

# Timeline

| Time | Event |
|------|-------|
| 14:00 | Increased traffic caused memory usage to rise. |
| 14:05 | Container exceeded its memory limit and was terminated (OOMKilled). |
| 14:06 | Kubernetes restarted the container automatically. |
| 14:10 - 14:40 | Continuous restart loop occurred due to repeated OOMKilled events. |
| 14:25 | Monitoring system generated high memory usage alerts. |
| 14:35 | SRE team began investigating pod events and resource metrics. |
| 14:45 | Memory limit was increased and the deployment was restarted. |
| 14:45 | Application recovered successfully. |

---

# Root Cause Analysis

### Primary Root Cause

The application exceeded the configured memory limit. When the container reached the limit, the Linux Out Of Memory (OOM) Killer terminated the process.

### Contributing Factors

- Memory limits were configured too low.
- No Horizontal Pod Autoscaler (HPA) was configured.
- Monitoring alerts were triggered after the application became unstable.
- No load testing had been performed before production deployment.

---

# Resolution

The following actions restored the service:

- Increased container memory limits.
- Restarted the deployment.
- Verified application health using `/health`.
- Confirmed stable memory usage.

---

# Preventive Actions

### Immediate

- Increase memory requests and limits.
- Add memory usage alerts.
- Perform load testing before deployment.

### Long-term

- Configure Horizontal Pod Autoscaler (HPA).
- Implement automatic scaling based on CPU and Memory.
- Enable centralized logging.
- Create SLOs and alerting rules.
- Review resource requests for every deployment.

---

# Auto-Scaling Policy

To prevent similar incidents:

- Minimum Replicas: **2**
- Maximum Replicas: **10**
- Target CPU Utilization: **70%**
- Target Memory Utilization: **75%**
- Scale Up immediately when thresholds are exceeded.
- Scale Down gradually after 5 minutes of stable utilization.
- Configure PodDisruptionBudget to maintain availability.

---

# Early Detection Using Ghaymah Monitoring

The issue can be detected early by monitoring:

- Container Memory Usage
- Pod Restart Count
- OOMKilled Events
- Container Health Checks
- Response Time
- Error Rate (5xx)
- Request Rate
- Application Availability

Recommended alerts:

- Memory Usage > 80%
- Restart Count > 3 within 5 minutes
- Health Check Failure
- Response Time > 500 ms
- Availability below 99%

---

# Lessons Learned

- Configure appropriate resource limits.
- Enable proactive monitoring and alerting.
- Test the application under expected production load.
- Implement autoscaling before deploying production workloads.
- Continuously review resource utilization trends.