4.0 KiB
Incident Postmortem Report
Incident Summary
On 27 July 2026, an application deployed on the Ghaymah platform experienced repeated OOMKilled events, resulting in approximately 45 minutes of downtime. During this period, users were unable to access the service because the application pods kept restarting after exceeding their memory limit.
Timeline
| Time | Event |
|---|---|
| 10:00 | A new application version was deployed. |
| 10:08 | Memory usage began increasing steadily. |
| 10:12 | The first pod was terminated with an OOMKilled event. |
| 10:15 | Kubernetes automatically restarted the pod. |
| 10:18 – 10:40 | The application entered a restart loop as multiple pods were repeatedly OOMKilled. |
| 10:42 | The SRE team investigated pod events, logs, and resource metrics. |
| 10:45 | Memory limits were increased temporarily, and the application's memory usage was reviewed. |
| 10:47 | A new healthy pod started successfully. |
| 10:55 | The service was fully restored and became available to users again. |
Root Cause
The application gradually consumed more memory than the Kubernetes memory limit that had been configured for the container. Once the limit was exceeded, Kubernetes terminated the container with an OOMKilled event.
Although Kubernetes restarted the pod automatically, the application continued to exceed its memory limit after each restart. This created a restart loop that kept the service unavailable until the memory issue was addressed.
Impact
- Approximately 45 minutes of service downtime.
- Multiple pod restarts due to repeated OOMKilled events.
- Users experienced failed requests and service unavailability.
- Increased operational effort to investigate and restore the application.
Resolution
The incident was resolved by:
- Investigating Kubernetes pod events and application logs.
- Identifying excessive memory consumption.
- Increasing the container memory limit temporarily.
- Restarting the affected pods.
- Verifying that the application returned to a healthy state.
Recommendations
To reduce the likelihood of similar incidents:
- Configure appropriate CPU and memory requests and limits.
- Enable Horizontal Pod Autoscaler (HPA).
- Create alerts for high memory utilization.
- Monitor pod restart counts and OOMKilled events.
- Perform load testing before deploying new releases.
- Review application memory usage regularly to identify potential memory leaks.
Auto-Scaling Policy for Ghaymah Platform
To prevent similar incidents, the platform should automatically scale the application based on resource utilization.
Horizontal Pod Autoscaler (HPA)
- Minimum replicas: 2
- Maximum replicas: 10
Scaling Rules
- Scale out when CPU utilization exceeds 70%.
- Scale out when memory utilization exceeds 75%.
- Scale in only after resource utilization remains below 40% for several minutes to avoid frequent scaling.
This policy helps distribute incoming traffic across multiple pods before any single container reaches its memory limit.
Early Detection Using Ghaymah Monitoring
The issue can be detected before it causes downtime by continuously monitoring application and Kubernetes metrics.
Key Metrics
- Container memory usage
- Memory utilization percentage
- CPU utilization
- Pod restart count
- OOMKilled events
- API response time
- HTTP 5xx error rate
- Application availability
Alerting Rules
Create alerts when:
- Memory utilization exceeds 80% for more than 5 minutes.
- A pod restarts more than 3 times within 10 minutes.
- An OOMKilled event occurs.
- API response time exceeds the defined threshold.
- Service availability drops below the expected level.
Dashboard
A monitoring dashboard should display:
- CPU usage
- Memory usage
- Running pods
- Pod restart count
- Request rate
- Response time
- Active alerts
With these metrics and alerts in place, the operations team can detect abnormal memory growth early and take corrective action before users are affected.