124 أسطر
4.0 KiB
Markdown
124 أسطر
4.0 KiB
Markdown
# Incident Postmortem Report
|
||
|
||
## Incident Summary
|
||
|
||
On 27 July 2026, an application deployed on the Ghaymah platform experienced repeated **OOMKilled** events, resulting in approximately **45 minutes of downtime**. During this period, users were unable to access the service because the application pods kept restarting after exceeding their memory limit.
|
||
|
||
---
|
||
|
||
# Timeline
|
||
|
||
| Time | Event |
|
||
|------|-------|
|
||
| 10:00 | A new application version was deployed. |
|
||
| 10:08 | Memory usage began increasing steadily. |
|
||
| 10:12 | The first pod was terminated with an **OOMKilled** event. |
|
||
| 10:15 | Kubernetes automatically restarted the pod. |
|
||
| 10:18 – 10:40 | The application entered a restart loop as multiple pods were repeatedly OOMKilled. |
|
||
| 10:42 | The SRE team investigated pod events, logs, and resource metrics. |
|
||
| 10:45 | Memory limits were increased temporarily, and the application's memory usage was reviewed. |
|
||
| 10:47 | A new healthy pod started successfully. |
|
||
| 10:55 | The service was fully restored and became available to users again. |
|
||
|
||
---
|
||
|
||
# Root Cause
|
||
|
||
The application gradually consumed more memory than the Kubernetes memory limit that had been configured for the container. Once the limit was exceeded, Kubernetes terminated the container with an **OOMKilled** event.
|
||
|
||
Although Kubernetes restarted the pod automatically, the application continued to exceed its memory limit after each restart. This created a restart loop that kept the service unavailable until the memory issue was addressed.
|
||
|
||
---
|
||
|
||
# Impact
|
||
|
||
- Approximately **45 minutes** of service downtime.
|
||
- Multiple pod restarts due to repeated **OOMKilled** events.
|
||
- Users experienced failed requests and service unavailability.
|
||
- Increased operational effort to investigate and restore the application.
|
||
|
||
---
|
||
|
||
# Resolution
|
||
|
||
The incident was resolved by:
|
||
|
||
- Investigating Kubernetes pod events and application logs.
|
||
- Identifying excessive memory consumption.
|
||
- Increasing the container memory limit temporarily.
|
||
- Restarting the affected pods.
|
||
- Verifying that the application returned to a healthy state.
|
||
|
||
---
|
||
|
||
# Recommendations
|
||
|
||
To reduce the likelihood of similar incidents:
|
||
|
||
- Configure appropriate CPU and memory requests and limits.
|
||
- Enable Horizontal Pod Autoscaler (HPA).
|
||
- Create alerts for high memory utilization.
|
||
- Monitor pod restart counts and OOMKilled events.
|
||
- Perform load testing before deploying new releases.
|
||
- Review application memory usage regularly to identify potential memory leaks.
|
||
|
||
---
|
||
|
||
# Auto-Scaling Policy for Ghaymah Platform
|
||
|
||
To prevent similar incidents, the platform should automatically scale the application based on resource utilization.
|
||
|
||
## Horizontal Pod Autoscaler (HPA)
|
||
|
||
- Minimum replicas: **2**
|
||
- Maximum replicas: **10**
|
||
|
||
### Scaling Rules
|
||
|
||
- Scale out when CPU utilization exceeds **70%**.
|
||
- Scale out when memory utilization exceeds **75%**.
|
||
- Scale in only after resource utilization remains below **40%** for several minutes to avoid frequent scaling.
|
||
|
||
This policy helps distribute incoming traffic across multiple pods before any single container reaches its memory limit.
|
||
|
||
---
|
||
|
||
# Early Detection Using Ghaymah Monitoring
|
||
|
||
The issue can be detected before it causes downtime by continuously monitoring application and Kubernetes metrics.
|
||
|
||
## Key Metrics
|
||
|
||
- Container memory usage
|
||
- Memory utilization percentage
|
||
- CPU utilization
|
||
- Pod restart count
|
||
- OOMKilled events
|
||
- API response time
|
||
- HTTP 5xx error rate
|
||
- Application availability
|
||
|
||
## Alerting Rules
|
||
|
||
Create alerts when:
|
||
|
||
- Memory utilization exceeds **80%** for more than 5 minutes.
|
||
- A pod restarts more than **3 times** within 10 minutes.
|
||
- An **OOMKilled** event occurs.
|
||
- API response time exceeds the defined threshold.
|
||
- Service availability drops below the expected level.
|
||
|
||
## Dashboard
|
||
|
||
A monitoring dashboard should display:
|
||
|
||
- CPU usage
|
||
- Memory usage
|
||
- Running pods
|
||
- Pod restart count
|
||
- Request rate
|
||
- Response time
|
||
- Active alerts
|
||
|
||
With these metrics and alerts in place, the operations team can detect abnormal memory growth early and take corrective action before users are affected.
|