# Ghaymah Postmortem Report

## Incident Summary

On July 27, 2026, the Ghaymah application experienced a service outage lasting approximately 45 minutes. The outage was caused by repeated OOMKilled events: the application's memory usage kept increasing until it hit the container's memory limit, causing the pod to crash and restart. Since the memory issue wasn't fixed, this kept happening in a loop until the memory limit was increased and the service stabilized.

**Severity:** SEV-2

---

## Impact

- The application was unavailable / responded with errors on and off for about 45 minutes.
- Requests failed whenever the pod was restarting.
- No data was lost.
- Service went back to normal after the memory limit was increased.

---

## Timeline

| Time | Event |
|------|-------|
| 12:00 | Application running normally. |
| 12:05 | Memory usage started increasing. |
| 12:06 | First OOMKilled event, pod restarted automatically. |
| 12:08 | Monitoring alert triggered (high memory / restart). |
| 12:10 – 12:40 | Memory kept climbing after each restart, causing repeated OOMKilled events. |
| 12:42 | Root cause identified (memory leak + low memory limit). |
| 12:45 | Memory limit increased and deployment restarted. |
| 12:50 | Service stable again. |

---

## Root Cause

The main cause was a memory leak in the application, which made memory usage grow continuously instead of staying stable. Because the memory limit set on the container was low compared to this growing usage, the limit was hit quickly and Kubernetes kept killing and restarting the pod (OOMKilled), which is why the outage lasted for several restart cycles instead of just one crash.

So: the leak was the actual bug, and the low memory limit is why it turned into a repeated 45-minute outage instead of a single failure.

---

## Recommendations

- Fix the memory leak in the application.
- Set a more realistic memory request/limit based on real usage, not a guess.
- Add an alert for high memory usage (e.g. 80%) before it reaches the limit.
- Consider auto-scaling so the app can handle load spikes without hitting the limit.
- Test the app under load before deploying, to catch this kind of issue earlier.

---

## Lessons Learned

- A memory leak can cause a much bigger outage than expected if the memory limit isn't sized correctly.
- Auto-restart helps keep the app "up," but it hides the real problem instead of fixing it.
- Alerting on memory usage trends (not just on crashes) would have caught this earlier.