5.9 KiB
Incident Postmortem Report
Application Information
| Item | Value |
|---|---|
| Application | API Service on Ghaymah Platform |
| Incident Date | 2026-07-27 |
| Incident Duration | 45 Minutes (10:15 – 11:00) |
| Severity | High |
| Impact | Complete service outage. Users were unable to access the API during the incident. |
1. Executive Summary
On 2026-07-27, the API service experienced a 45-minute outage due to repeated OOMKilled events.
The application gradually consumed more memory because of a memory leak while processing large requests. Once the container exceeded its allocated memory limit, the platform terminated and restarted it automatically. This restart cycle repeated continuously, resulting in complete service unavailability.
The immediate mitigation was increasing the container memory allocation and restarting the service. Long-term corrective actions include fixing the memory leak, improving monitoring, and implementing an auto-scaling policy based on resource utilization.
2. Incident Timeline
| Time | Event |
|---|---|
| 10:15 | Memory usage started increasing on one of the application containers. |
| 10:22 | First container was terminated with OOMKilled and automatically restarted by the platform. |
| 10:25 – 10:45 | Multiple containers repeatedly entered restart loops, reducing the application's availability. |
| 10:45 | Monitoring system generated alerts indicating complete service outage. |
| 10:50 | SRE team began investigating the incident and identified abnormal memory usage. |
| 11:00 | Container memory limit increased from 512 MiB to 1 GiB and services were restarted successfully. |
| 11:15 | Additional monitoring and temporary mitigation measures were applied. |
3. Root Cause Analysis
Immediate Cause
The application exhausted the available memory, causing the platform to terminate the container with an OOMKilled event.
Root Cause
A memory leak inside the request processing logic (or an external library) continuously retained memory instead of releasing it after requests were completed.
Contributing Factors
- Sudden increase in traffic (approximately 3× normal load).
- Insufficient memory allocation.
- No automatic scaling based on memory utilization.
- Lack of early warning alerts before reaching the memory limit.
4. Recommendations
Short-Term Actions
- Increase container memory limits.
- Monitor memory consumption continuously.
- Restart unhealthy containers automatically.
Long-Term Improvements
- Fix the memory leak in the application.
- Configure appropriate container memory requests and limits.
- Enable memory-based auto-scaling.
- Configure proactive alerting.
- Perform regular load and stress testing.
- Maintain an incident response runbook for OOMKilled events.
5. Auto-Scaling Policy
To prevent similar incidents, the following auto-scaling policy is recommended.
| Configuration | Value |
|---|---|
| Metric | Average Container Memory Usage |
| Scale-Out Threshold | Memory usage > 70% for 2 minutes |
| Scale-Out Action | Add one container instance |
| Minimum Replicas | 2 |
| Maximum Replicas | 10 |
| Cooldown Period | 5 Minutes |
| Scale-In Threshold | Memory usage < 40% for 10 minutes |
| Scale-In Action | Remove one container instance |
Example Auto-Scaling Configuration
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
name: api-hpa
spec:
minReplicas: 2
maxReplicas: 10
metrics:
- type: Resource
resource:
name: memory
target:
type: Utilization
averageUtilization: 70
Note: The YAML above is an example of a memory-based auto-scaling policy and illustrates the desired behavior conceptually.
6. Early Detection Using Ghaymah Monitoring
The following monitoring capabilities should be configured to detect similar incidents before they impact users.
Resource Monitoring
Track real-time metrics including:
- Container Memory Usage
- CPU Utilization
- Container Restart Count
- Container Health Status
Alerting
Configure alerts such as:
| Condition | Action |
|---|---|
| Memory usage > 60% for 3 minutes | Send Email / Slack / Webhook notification |
| Multiple container restarts | Trigger High-Priority Alert |
| Service unavailable | Immediate Critical Alert |
Event Monitoring
Monitor platform events including:
- OOMKilled
- Container Restart
- CrashLoop
- Failed Deployments
Dashboard
Create a monitoring dashboard displaying:
- Current Container Status
- Memory Utilization
- CPU Utilization
- Response Time
- Request Count
- Restart Count
- OOMKilled Events
- Service Availability (Uptime)
Trend Analysis
Analyze historical metrics to identify:
- Gradual memory growth
- Traffic spikes
- Abnormal restart frequency
- Resource utilization trends
This enables proactive detection before a service outage occurs.
7. Lessons Learned
The incident highlighted several important operational improvements:
- Proper memory sizing is essential.
- Continuous monitoring should be proactive rather than reactive.
- Memory-based auto-scaling reduces service interruption.
- Early alerts significantly reduce recovery time.
- Load testing helps identify memory leaks before production deployment.
Action Items
| Priority | Action | Status |
|---|---|---|
| High | Fix memory leak | Planned |
| High | Configure memory alerts | Planned |
| High | Enable auto-scaling | Planned |
| Medium | Perform load testing | Planned |
| Medium | Update operational runbook | Planned |
Report Information
| Item | Value |
|---|---|
| Prepared By | SRE Team |
| Platform | Ghaymah Systems |
| Report Date | 2026-07-27 |
| Incident Type | Repeated OOMKilled |
| Status | Closed |