# Incident Postmortem: Application Frequent Crashes (OOMKilled)

---

## 1. Incident Overview & Summary
- **Incident Date:** July 26, 2026
- **Impact:** The application deployed on Ghaymah Cloud suffered from repeated crashes every 45 minutes, causing service downtime for users.
- **Root Cause:** **OOMKilled (Out Of Memory)** error triggered because the container exceeded its allocated memory limit due to unoptimized resource configuration or a potential memory leak.

---

## 2. Timeline
- **00:00** - Application deployed and running normally.
- **00:30** - Memory consumption gradually increases as traffic and background tasks accumulate.
- **00:45** - Memory usage reaches 100% of the container limit; the Linux kernel terminates the process (`OOMKilled`).
- **00:46** - Container restarts automatically, and the cycle repeats precisely every 45 minutes.

---

## 3. Root Cause Analysis
1. **Resource Limits:** The container was assigned insufficient RAM limits in its deployment configuration.
2. **Memory Leak / Garbage Collection:** Long-running background jobs or unclosed database connections failed to release memory back to the system, leading to a steady upward trend until exhaustion.

---

## 4. Preventive Actions & Recommendations
- **Increase Container RAM:** Scale up the memory limits in the deployment configuration on Ghaymah Cloud.
- **Code Optimization:** Audit the application code to fix any memory leaks and ensure proper resource disposal.
- **Health & Memory Probes:** Implement stricter liveness and readiness probes.

---

## 5. Auto-Scaling Policy Design
To prevent recurring downtime under high load, an auto-scaling policy should be configured on Ghaymah Cloud:
- **Trigger Metric:** Memory Utilization or CPU Utilization.
- **Scale-Up Threshold:** Automatically add a new container instance (replica) when memory usage exceeds **75%** for 5 consecutive minutes.
- **Max Instances:** Set a cap of 3 instances to handle traffic spikes safely without exhausting cluster resources.

---

## 6. Early Detection & Monitoring Setup
Using Ghaymah Cloud monitoring tools to detect the issue proactively before crashes occur:
- **Custom Alerts:** Set up an alerting rule (via Webhook/Email) to notify the DevOps team when container memory usage crosses **70%**.
- **Real-Time Dashboards:** Monitor memory trends continuously using the metrics dashboard to spot gradual upward slopes before hitting the OOM threshold.