44 أسطر
2.4 KiB
Markdown
44 أسطر
2.4 KiB
Markdown
# Incident Postmortem: Application Frequent Crashes (OOMKilled)
|
|
|
|
---
|
|
|
|
## 1. Incident Overview & Summary
|
|
- **Incident Date:** July 26, 2026
|
|
- **Impact:** The application deployed on Ghaymah Cloud suffered from repeated crashes every 45 minutes, causing service downtime for users.
|
|
- **Root Cause:** **OOMKilled (Out Of Memory)** error triggered because the container exceeded its allocated memory limit due to unoptimized resource configuration or a potential memory leak.
|
|
|
|
---
|
|
|
|
## 2. Timeline
|
|
- **00:00** - Application deployed and running normally.
|
|
- **00:30** - Memory consumption gradually increases as traffic and background tasks accumulate.
|
|
- **00:45** - Memory usage reaches 100% of the container limit; the Linux kernel terminates the process (`OOMKilled`).
|
|
- **00:46** - Container restarts automatically, and the cycle repeats precisely every 45 minutes.
|
|
|
|
---
|
|
|
|
## 3. Root Cause Analysis
|
|
1. **Resource Limits:** The container was assigned insufficient RAM limits in its deployment configuration.
|
|
2. **Memory Leak / Garbage Collection:** Long-running background jobs or unclosed database connections failed to release memory back to the system, leading to a steady upward trend until exhaustion.
|
|
|
|
---
|
|
|
|
## 4. Preventive Actions & Recommendations
|
|
- **Increase Container RAM:** Scale up the memory limits in the deployment configuration on Ghaymah Cloud.
|
|
- **Code Optimization:** Audit the application code to fix any memory leaks and ensure proper resource disposal.
|
|
- **Health & Memory Probes:** Implement stricter liveness and readiness probes.
|
|
|
|
---
|
|
|
|
## 5. Auto-Scaling Policy Design
|
|
To prevent recurring downtime under high load, an auto-scaling policy should be configured on Ghaymah Cloud:
|
|
- **Trigger Metric:** Memory Utilization or CPU Utilization.
|
|
- **Scale-Up Threshold:** Automatically add a new container instance (replica) when memory usage exceeds **75%** for 5 consecutive minutes.
|
|
- **Max Instances:** Set a cap of 3 instances to handle traffic spikes safely without exhausting cluster resources.
|
|
|
|
---
|
|
|
|
## 6. Early Detection & Monitoring Setup
|
|
Using Ghaymah Cloud monitoring tools to detect the issue proactively before crashes occur:
|
|
- **Custom Alerts:** Set up an alerting rule (via Webhook/Email) to notify the DevOps team when container memory usage crosses **70%**.
|
|
- **Real-Time Dashboards:** Monitor memory trends continuously using the metrics dashboard to spot gradual upward slopes before hitting the OOM threshold. |