388 أسطر
17 KiB
Markdown
388 أسطر
17 KiB
Markdown
# 🔴 Incident Postmortem Report — OOMKilled Application Crash
|
||
|
||
**Incident ID:** INC-2026-0726
|
||
**Severity:** P1 — Critical
|
||
**Duration:** 45 minutes
|
||
**Date:** July 26, 2026
|
||
**Author:** DevOps Team
|
||
**Status:** Resolved
|
||
|
||
---
|
||
|
||
## 1. Executive Summary
|
||
|
||
On July 26, 2026, the **Ghayma REST API** application hosted on a cloud container platform experienced a **45-minute outage** caused by repeated **OOMKilled** (Out of Memory Killed) events. The container runtime terminated the application process multiple times after it exceeded its allocated memory limit. Each restart triggered the same memory spike, creating a crash loop that prevented the service from recovering.
|
||
|
||
**Impact:**
|
||
- 100% service unavailability for 45 minutes
|
||
- All API consumers (frontend clients, monitoring, health probes) received connection errors
|
||
- Approximately **2,700 failed requests** during the outage window (estimated ~60 req/min baseline)
|
||
- Container orchestrator marked the pod/container as `CrashLoopBackOff` after repeated restart failures
|
||
|
||
---
|
||
|
||
## 2. Incident Timeline
|
||
|
||
| Time (UTC) | Event |
|
||
| :--- | :--- |
|
||
| **11:00** | 🟢 Application running normally. Memory usage stable at ~120MB (limit: 256MB) |
|
||
| **11:12** | 📈 Traffic spike begins — external batch job sends large payload requests to `POST /api/v1/items` |
|
||
| **11:15** | ⚠️ Memory usage crosses **200MB** (78% of limit). No alerts triggered |
|
||
| **11:18** | 🔴 Memory hits **256MB** limit. Kernel OOM killer terminates the container process (`OOMKilled`) |
|
||
| **11:18** | 🔄 Container runtime automatically restarts the container (Restart #1) |
|
||
| **11:20** | 🔴 Application starts, loads cached data into memory, immediately OOMKilled again (Restart #2) |
|
||
| **11:20–11:45** | 🔁 **Crash loop** — container restarts 12 times. Orchestrator applies exponential backoff (`CrashLoopBackOff`) |
|
||
| **11:32** | 📟 On-call engineer alerted via PagerDuty after health check failures exceed 10 minutes |
|
||
| **11:38** | 🔍 Engineer identifies OOMKilled events in container logs and platform event stream |
|
||
| **11:45** | 🛠️ Engineer increases memory limit from **256MB → 512MB** and deploys hotfix |
|
||
| **11:48** | 🔄 Application restarts successfully. Memory stabilizes at ~180MB |
|
||
| **11:50** | 📉 Batch job completes. Traffic returns to normal levels |
|
||
| **12:03** | 🟢 Service fully confirmed stable. Incident resolved |
|
||
|
||
**Total downtime:** 45 minutes (11:18 — 12:03 UTC)
|
||
|
||
---
|
||
|
||
## 3. Root Cause Analysis
|
||
|
||
### 3.1 Direct Cause
|
||
|
||
The container was configured with a **memory limit of 256MB**, which was insufficient to handle traffic spikes. When an external batch job sent a burst of large `POST` requests with sizable JSON payloads, the Node.js process accumulated in-memory data (parsed request bodies, in-memory item storage, response buffers) beyond the container's limit.
|
||
|
||
The Linux kernel's **OOM Killer** terminated the process when it attempted to allocate memory beyond the `256MB` cgroup limit.
|
||
|
||
### 3.2 Contributing Factors
|
||
|
||
```
|
||
┌──────────────────────────────────────────────────────────────────┐
|
||
│ ROOT CAUSE BREAKDOWN │
|
||
├──────────────────────────────────────────────────────────────────┤
|
||
│ │
|
||
│ 1. INSUFFICIENT MEMORY LIMIT │
|
||
│ └─ Container limited to 256MB (too tight for Node.js) │
|
||
│ │
|
||
│ 2. NO MEMORY-AWARE AUTO-SCALING │
|
||
│ └─ Only 1 replica running, no HPA/scaling policy │
|
||
│ │
|
||
│ 3. UNBOUNDED IN-MEMORY DATA STORE │
|
||
│ └─ Items array grows indefinitely with no cap │
|
||
│ │
|
||
│ 4. NO MEMORY ALERTS / EARLY WARNING │
|
||
│ └─ Alert only triggered after 10 min of health failures │
|
||
│ │
|
||
│ 5. NO REQUEST PAYLOAD SIZE LIMIT │
|
||
│ └─ Express accepted arbitrarily large JSON bodies │
|
||
│ │
|
||
└──────────────────────────────────────────────────────────────────┘
|
||
```
|
||
|
||
### 3.3 Why Did Restarts Fail to Recover?
|
||
|
||
Each restart reloaded the same conditions:
|
||
1. Container starts → Node.js initializes (~80MB baseline)
|
||
2. Pending queued requests immediately hit the API
|
||
3. Large payloads + in-memory storage push memory past the limit
|
||
4. OOMKilled again within seconds
|
||
5. Orchestrator enters **CrashLoopBackOff** with increasing delays between restarts
|
||
|
||
---
|
||
|
||
## 4. Remediation Actions Taken
|
||
|
||
| Action | Status |
|
||
| :--- | :--- |
|
||
| Increased container memory limit to 512MB | ✅ Done (hotfix) |
|
||
| Restarted container with new limits | ✅ Done |
|
||
| Confirmed service recovery and stability | ✅ Done |
|
||
|
||
---
|
||
|
||
## 5. Recommendations & Prevention
|
||
|
||
### 5.1 Application-Level Fixes
|
||
|
||
#### A. Limit Request Payload Size
|
||
```javascript
|
||
// In src/index.js — restrict incoming JSON body size
|
||
app.use(express.json({ limit: '1mb' }));
|
||
```
|
||
> Prevents a single large request from consuming excessive memory.
|
||
|
||
#### B. Cap In-Memory Data Store
|
||
```javascript
|
||
// Limit the items array to prevent unbounded growth
|
||
const MAX_ITEMS = 1000;
|
||
|
||
app.post('/api/v1/items', (req, res) => {
|
||
if (items.length >= MAX_ITEMS) {
|
||
return res.status(429).json({
|
||
success: false,
|
||
error: 'Maximum item limit reached'
|
||
});
|
||
}
|
||
// ... rest of handler
|
||
});
|
||
```
|
||
|
||
#### C. Set Node.js Memory Ceiling
|
||
```dockerfile
|
||
# In Dockerfile — set explicit V8 heap limit
|
||
CMD ["node", "--max-old-space-size=384", "src/index.js"]
|
||
```
|
||
> Ensures Node.js garbage collector runs more aggressively before hitting the container limit.
|
||
|
||
#### D. Use External Storage for Production
|
||
Replace the in-memory `items` array with a proper database (Redis, PostgreSQL, MongoDB) so application memory remains constant regardless of data volume.
|
||
|
||
---
|
||
|
||
### 5.2 Container & Infrastructure Fixes
|
||
|
||
#### A. Set Proper Resource Requests & Limits
|
||
|
||
```yaml
|
||
# Kubernetes Deployment example
|
||
resources:
|
||
requests:
|
||
memory: "256Mi" # Guaranteed minimum
|
||
cpu: "100m"
|
||
limits:
|
||
memory: "512Mi" # Hard ceiling
|
||
cpu: "500m"
|
||
```
|
||
|
||
> [!IMPORTANT]
|
||
> **Rule of thumb:** Set `limits.memory` to at least **2x** the average working set. Set `requests.memory` to the **steady-state average**.
|
||
|
||
#### B. Add Graceful Shutdown Handling
|
||
```javascript
|
||
process.on('SIGTERM', () => {
|
||
console.log('SIGTERM received. Shutting down gracefully...');
|
||
server.close(() => {
|
||
process.exit(0);
|
||
});
|
||
});
|
||
```
|
||
|
||
---
|
||
|
||
## 6. Auto-Scaling Policy Design
|
||
|
||
### 6.1 Horizontal Pod Autoscaler (HPA) Policy
|
||
|
||
The following auto-scaling policy prevents a single container from being overwhelmed by distributing load across multiple replicas:
|
||
|
||
```yaml
|
||
apiVersion: autoscaling/v2
|
||
kind: HorizontalPodAutoscaler
|
||
metadata:
|
||
name: ghayma-api-hpa
|
||
namespace: production
|
||
spec:
|
||
scaleTargetRef:
|
||
apiVersion: apps/v1
|
||
kind: Deployment
|
||
name: ghayma-api
|
||
|
||
# Replica bounds
|
||
minReplicas: 2 # Always run at least 2 for high availability
|
||
maxReplicas: 10 # Cap to control costs
|
||
|
||
# Scaling metrics
|
||
metrics:
|
||
# Scale based on MEMORY usage (primary - prevents OOMKill)
|
||
- type: Resource
|
||
resource:
|
||
name: memory
|
||
target:
|
||
type: Utilization
|
||
averageUtilization: 70 # Scale up when memory > 70% of limit
|
||
|
||
# Scale based on CPU usage (secondary)
|
||
- type: Resource
|
||
resource:
|
||
name: cpu
|
||
target:
|
||
type: Utilization
|
||
averageUtilization: 75 # Scale up when CPU > 75%
|
||
|
||
# Scaling behavior (prevents flapping)
|
||
behavior:
|
||
scaleUp:
|
||
stabilizationWindowSeconds: 30 # React quickly to spikes
|
||
policies:
|
||
- type: Pods
|
||
value: 2 # Add up to 2 pods at a time
|
||
periodSeconds: 60
|
||
scaleDown:
|
||
stabilizationWindowSeconds: 300 # Wait 5 min before scaling down
|
||
policies:
|
||
- type: Pods
|
||
value: 1 # Remove 1 pod at a time
|
||
periodSeconds: 120
|
||
```
|
||
|
||
### 6.2 How This Policy Prevents Duplication of the Incident
|
||
|
||
```
|
||
Traffic Spike Detected
|
||
│
|
||
▼
|
||
┌───────────────────────┐
|
||
│ Memory usage > 70% │──── YES ──▶ HPA adds new replicas (up to 10)
|
||
│ on existing pods? │ Load balancer distributes traffic
|
||
└───────────────────────┘ Memory per-pod stays under limit
|
||
│ NO │
|
||
▼ ▼
|
||
Normal operation Spike absorbed across multiple pods
|
||
No single pod reaches OOMKill threshold
|
||
```
|
||
|
||
### 6.3 Cloud Platform Auto-Scaling (Non-Kubernetes)
|
||
|
||
For managed container platforms (e.g., Ghayma Cloud, AWS ECS, Google Cloud Run):
|
||
|
||
| Setting | Value | Rationale |
|
||
| :--- | :--- | :--- |
|
||
| **Min instances** | 2 | Avoid cold starts & single point of failure |
|
||
| **Max instances** | 10 | Cost ceiling |
|
||
| **Scale-up trigger** | Memory > 70% OR CPU > 75% OR Concurrent requests > 50 | Multi-signal scaling |
|
||
| **Scale-up cooldown** | 30 seconds | React quickly to spikes |
|
||
| **Scale-down cooldown** | 5 minutes | Prevent flapping during variable traffic |
|
||
| **Memory per instance** | 512MB | 2x steady-state working set |
|
||
|
||
---
|
||
|
||
## 7. Early Detection — Monitoring & Alerting Strategy
|
||
|
||
### 7.1 Key Metrics to Monitor
|
||
|
||
| Metric | Source | Warning Threshold | Critical Threshold |
|
||
| :--- | :--- | :--- | :--- |
|
||
| **Container Memory Usage** | cAdvisor / Platform metrics | > 70% of limit | > 85% of limit |
|
||
| **Container Restart Count** | Kubelet / Platform events | ≥ 1 restart in 5 min | ≥ 3 restarts in 10 min |
|
||
| **OOMKilled Events** | Kernel / Container runtime | Any occurrence | N/A (always critical) |
|
||
| **Response Time (p95)** | Application `/metrics` endpoint | > 500ms | > 2000ms |
|
||
| **Error Rate (5xx)** | Load balancer / Application | > 1% | > 5% |
|
||
| **Health Check Failures** | Platform health probe | 1 consecutive failure | 3 consecutive failures |
|
||
| **Request Queue Depth** | Load balancer | > 50 pending | > 200 pending |
|
||
|
||
### 7.2 Alerting Rules (Prometheus / Grafana Example)
|
||
|
||
```yaml
|
||
# Alert: Memory approaching container limit
|
||
- alert: HighMemoryUsage
|
||
expr: |
|
||
(container_memory_usage_bytes{container="ghayma-api"}
|
||
/ container_spec_memory_limit_bytes{container="ghayma-api"}) > 0.70
|
||
for: 2m
|
||
labels:
|
||
severity: warning
|
||
annotations:
|
||
summary: "Ghayma API memory usage above 70%"
|
||
description: "Container {{ $labels.pod }} memory at {{ $value | humanizePercentage }}"
|
||
|
||
# Alert: OOMKill detected (CRITICAL — immediate page)
|
||
- alert: OOMKillDetected
|
||
expr: |
|
||
increase(kube_pod_container_status_restarts_total{container="ghayma-api"}[5m]) > 0
|
||
and kube_pod_container_status_last_terminated_reason{container="ghayma-api"} == "OOMKilled"
|
||
for: 0m
|
||
labels:
|
||
severity: critical
|
||
annotations:
|
||
summary: "OOMKill detected on Ghayma API"
|
||
description: "Pod {{ $labels.pod }} was OOMKilled. Immediate investigation required."
|
||
|
||
# Alert: High error rate
|
||
- alert: HighErrorRate
|
||
expr: |
|
||
rate(http_requests_total{status=~"5.."}[5m])
|
||
/ rate(http_requests_total[5m]) > 0.05
|
||
for: 3m
|
||
labels:
|
||
severity: critical
|
||
annotations:
|
||
summary: "Ghayma API error rate exceeds 5%"
|
||
```
|
||
|
||
### 7.3 Monitoring Architecture
|
||
|
||
```
|
||
┌─────────────────────────────────────────────────────────────────┐
|
||
│ MONITORING STACK │
|
||
│ │
|
||
│ ┌──────────┐ ┌─────────────┐ ┌──────────────────────┐ │
|
||
│ │ Ghayma │───▶│ Prometheus │───▶│ Grafana Dashboard │ │
|
||
│ │ API │ │ (scrapes │ │ (visualization + │ │
|
||
│ │ /metrics │ │ /metrics) │ │ alerting rules) │ │
|
||
│ └──────────┘ └──────┬──────┘ └──────────────────────┘ │
|
||
│ │ │
|
||
│ ▼ │
|
||
│ ┌─────────────────┐ │
|
||
│ │ Alert Manager │ │
|
||
│ │ (routing & │ │
|
||
│ │ deduplication)│ │
|
||
│ └────────┬────────┘ │
|
||
│ │ │
|
||
│ ┌────────────┼─────────────┐ │
|
||
│ ▼ ▼ ▼ │
|
||
│ ┌──────────┐ ┌──────────┐ ┌───────────┐ │
|
||
│ │ Slack │ │ PagerDuty│ │ Email │ │
|
||
│ │ #alerts │ │ on-call │ │ team │ │
|
||
│ └──────────┘ └──────────┘ └───────────┘ │
|
||
└─────────────────────────────────────────────────────────────────┘
|
||
```
|
||
|
||
### 7.4 Cloud Platform Native Tools
|
||
|
||
| Cloud Platform | Monitoring Service | Key Feature for OOM Detection |
|
||
| :--- | :--- | :--- |
|
||
| **AWS ECS / Fargate** | CloudWatch Container Insights | `MemoryUtilization` metric + CloudWatch Alarms |
|
||
| **Google Cloud Run** | Cloud Monitoring | `container/memory/utilization` + Alert Policies |
|
||
| **Azure Container Apps** | Azure Monitor | `MemoryWorkingSet` + Metric Alerts |
|
||
| **Kubernetes (any)** | Prometheus + Grafana | `container_memory_usage_bytes` + `kube_pod_container_status_last_terminated_reason` |
|
||
|
||
### 7.5 Recommended Grafana Dashboard Panels
|
||
|
||
1. **Memory Usage vs Limit** — Time series showing memory consumption relative to the container limit (with 70% and 85% threshold lines)
|
||
2. **Container Restarts** — Counter panel showing restart events over time
|
||
3. **Response Time Heatmap** — p50 / p95 / p99 latency distribution
|
||
4. **Request Rate & Error Rate** — Dual-axis graph (total requests + 5xx rate)
|
||
5. **Pod/Replica Count** — Shows HPA scaling activity over time
|
||
|
||
---
|
||
|
||
## 8. Lessons Learned
|
||
|
||
| # | Lesson | Action Item |
|
||
| :--- | :--- | :--- |
|
||
| 1 | Default memory limits were set too low without load testing | Conduct load tests before production deployment |
|
||
| 2 | Single-replica deployment has no resilience | Always run **≥ 2 replicas** for production services |
|
||
| 3 | Alert threshold (10 min) was too slow for a P1 outage | Reduce critical alert threshold to **2 minutes** |
|
||
| 4 | In-memory data stores are dangerous without bounds | Use external databases or enforce collection size limits |
|
||
| 5 | No runbook existed for OOMKilled incidents | Create and publish an OOMKilled response runbook |
|
||
|
||
---
|
||
|
||
## 9. Action Items Tracker
|
||
|
||
| # | Action Item | Owner | Priority | Deadline | Status |
|
||
| :--- | :--- | :--- | :--- | :--- | :--- |
|
||
| 1 | Increase container memory limit to 512MB | DevOps | P0 | Done | ✅ |
|
||
| 2 | Add `express.json({ limit: '1mb' })` payload limit | Backend | P1 | +1 day | ⬜ |
|
||
| 3 | Implement HPA with memory-based scaling | DevOps | P1 | +3 days | ⬜ |
|
||
| 4 | Set up Prometheus memory alerts (70% / 85%) | DevOps | P1 | +3 days | ⬜ |
|
||
| 5 | Add OOMKilled alert rule (immediate paging) | DevOps | P0 | +1 day | ⬜ |
|
||
| 6 | Run minimum 2 replicas in production | DevOps | P1 | +1 day | ⬜ |
|
||
| 7 | Replace in-memory store with Redis/DB | Backend | P2 | +1 week | ⬜ |
|
||
| 8 | Conduct load testing with realistic traffic | QA | P2 | +2 weeks | ⬜ |
|
||
| 9 | Create OOMKilled incident runbook | DevOps | P2 | +1 week | ⬜ |
|
||
| 10 | Build Grafana dashboard for container metrics | DevOps | P2 | +1 week | ⬜ |
|
||
|
||
---
|
||
|
||
> **Report prepared by:** DevOps Team
|
||
> **Review date:** July 26, 2026
|
||
> **Next review:** August 2, 2026 (verify all P0/P1 items completed)
|