2.4 KiB
2.4 KiB
Incident Postmortem: Application Frequent Crashes (OOMKilled)
1. Incident Overview & Summary
- Incident Date: July 26, 2026
- Impact: The application deployed on Ghaymah Cloud suffered from repeated crashes every 45 minutes, causing service downtime for users.
- Root Cause: OOMKilled (Out Of Memory) error triggered because the container exceeded its allocated memory limit due to unoptimized resource configuration or a potential memory leak.
2. Timeline
- 00:00 - Application deployed and running normally.
- 00:30 - Memory consumption gradually increases as traffic and background tasks accumulate.
- 00:45 - Memory usage reaches 100% of the container limit; the Linux kernel terminates the process (
OOMKilled). - 00:46 - Container restarts automatically, and the cycle repeats precisely every 45 minutes.
3. Root Cause Analysis
- Resource Limits: The container was assigned insufficient RAM limits in its deployment configuration.
- Memory Leak / Garbage Collection: Long-running background jobs or unclosed database connections failed to release memory back to the system, leading to a steady upward trend until exhaustion.
4. Preventive Actions & Recommendations
- Increase Container RAM: Scale up the memory limits in the deployment configuration on Ghaymah Cloud.
- Code Optimization: Audit the application code to fix any memory leaks and ensure proper resource disposal.
- Health & Memory Probes: Implement stricter liveness and readiness probes.
5. Auto-Scaling Policy Design
To prevent recurring downtime under high load, an auto-scaling policy should be configured on Ghaymah Cloud:
- Trigger Metric: Memory Utilization or CPU Utilization.
- Scale-Up Threshold: Automatically add a new container instance (replica) when memory usage exceeds 75% for 5 consecutive minutes.
- Max Instances: Set a cap of 3 instances to handle traffic spikes safely without exhausting cluster resources.
6. Early Detection & Monitoring Setup
Using Ghaymah Cloud monitoring tools to detect the issue proactively before crashes occur:
- Custom Alerts: Set up an alerting rule (via Webhook/Email) to notify the DevOps team when container memory usage crosses 70%.
- Real-Time Dashboards: Monitor memory trends continuously using the metrics dashboard to spot gradual upward slopes before hitting the OOM threshold.