# Postmortem Report

## Incident Summary

**Incident:** Application outage due to repeated OOMKilled events.

**Duration:** 45 minutes

**Impact:**
- The application was unavailable for users.
- API requests failed during the outage.
- User experience was significantly affected.

---

## Timeline

| Time | Event |
|------|-------|
| 10:00 | Application deployed successfully. |
| 10:08 | Memory usage started increasing rapidly. |
| 10:12 | First OOMKilled event occurred. |
| 10:13 | Kubernetes restarted the container. |
| 10:15–10:40 | Continuous OOMKilled restart loop. |
| 10:42 | Engineering team investigated the issue. |
| 10:45 | Memory limit increased and application stabilized. |

---

## Root Cause

The application exceeded its allocated memory limit.

The container was repeatedly terminated by Kubernetes with an **OOMKilled** event because memory consumption continued growing beyond the configured limit.

Possible contributing factors:

- Memory leak inside the application.
- Memory limits configured too low.
- No Horizontal Pod Autoscaler.
- Lack of memory usage alerts.

---

## Resolution

The engineering team:

- Increased container memory limits.
- Restarted the deployment.
- Verified application health.
- Monitored memory consumption until stable.

---

## Recommendations

### Immediate Actions

- Configure appropriate memory requests and limits.
- Investigate memory leaks.
- Enable memory monitoring.
- Configure alerting.

### Long-Term Improvements

- Implement Horizontal Pod Autoscaler (HPA).
- Perform load testing before production.
- Review memory usage after each deployment.
- Add automatic scaling policies.
- Conduct regular post-deployment monitoring.

---

# Auto-Scaling Policy

To prevent similar incidents, the platform should implement:

- Minimum replicas: **2**
- Maximum replicas: **10**
- Scale out when:
  - Memory usage > 70%
  - CPU usage > 70%
- Scale in when:
  - Memory usage < 40%
  - CPU usage < 40%
- Cooldown period: **5 minutes**
- Enable automatic replacement of unhealthy pods.

This policy ensures enough capacity during traffic spikes while reducing resource waste during normal operation.

---

# Early Detection Using Ghaymah Monitoring

The issue can be detected early by monitoring:

- Memory usage
- CPU usage
- Container restart count
- OOMKilled events
- Pod health status
- Response time
- Error rate

Recommended alerts:

- Memory usage above 80%
- More than 3 container restarts within 10 minutes
- Pod enters CrashLoopBackOff
- Health endpoint becomes unavailable
- Response time exceeds acceptable thresholds

These monitoring practices allow engineers to respond before users experience service interruption.

---

# Lessons Learned

- Proper resource limits are essential.
- Continuous monitoring is critical.
- Autoscaling improves application availability.
- Early alerting reduces downtime.
- Capacity planning should be reviewed before production deployments.