الملفات

Anass_Ehab33 01b477b139 feat: Ghayma Assessment - complete submission

2026-07-26 22:52:49 +03:00

17 KiB

خام اللوم التاريخ

🔴 Incident Postmortem Report — OOMKilled Application Crash

Incident ID: INC-2026-0726
Severity: P1 — Critical
Duration: 45 minutes
Date: July 26, 2026
Author: DevOps Team
Status: Resolved

1. Executive Summary

On July 26, 2026, the Ghayma REST API application hosted on a cloud container platform experienced a 45-minute outage caused by repeated OOMKilled (Out of Memory Killed) events. The container runtime terminated the application process multiple times after it exceeded its allocated memory limit. Each restart triggered the same memory spike, creating a crash loop that prevented the service from recovering.

Impact:

100% service unavailability for 45 minutes
All API consumers (frontend clients, monitoring, health probes) received connection errors
Approximately 2,700 failed requests during the outage window (estimated ~60 req/min baseline)
Container orchestrator marked the pod/container as CrashLoopBackOff after repeated restart failures

2. Incident Timeline

Time (UTC)	Event
11:00	🟢 Application running normally. Memory usage stable at ~120MB (limit: 256MB)
11:12	📈 Traffic spike begins — external batch job sends large payload requests to `POST /api/v1/items`
11:15	⚠️ Memory usage crosses 200MB (78% of limit). No alerts triggered
11:18	🔴 Memory hits 256MB limit. Kernel OOM killer terminates the container process (`OOMKilled`)
11:18	🔄 Container runtime automatically restarts the container (Restart #1)
11:20	🔴 Application starts, loads cached data into memory, immediately OOMKilled again (Restart #2)
11:20–11:45	🔁 Crash loop — container restarts 12 times. Orchestrator applies exponential backoff (`CrashLoopBackOff`)
11:32	📟 On-call engineer alerted via PagerDuty after health check failures exceed 10 minutes
11:38	🔍 Engineer identifies OOMKilled events in container logs and platform event stream
11:45	🛠️ Engineer increases memory limit from 256MB → 512MB and deploys hotfix
11:48	🔄 Application restarts successfully. Memory stabilizes at ~180MB
11:50	📉 Batch job completes. Traffic returns to normal levels
12:03	🟢 Service fully confirmed stable. Incident resolved

Total downtime: 45 minutes (11:18 — 12:03 UTC)

3. Root Cause Analysis

3.1 Direct Cause

The container was configured with a memory limit of 256MB, which was insufficient to handle traffic spikes. When an external batch job sent a burst of large POST requests with sizable JSON payloads, the Node.js process accumulated in-memory data (parsed request bodies, in-memory item storage, response buffers) beyond the container's limit.

The Linux kernel's OOM Killer terminated the process when it attempted to allocate memory beyond the 256MB cgroup limit.

3.2 Contributing Factors

┌──────────────────────────────────────────────────────────────────┐
│                     ROOT CAUSE BREAKDOWN                        │
├──────────────────────────────────────────────────────────────────┤
│                                                                  │
│  1. INSUFFICIENT MEMORY LIMIT                                    │
│     └─ Container limited to 256MB (too tight for Node.js)       │
│                                                                  │
│  2. NO MEMORY-AWARE AUTO-SCALING                                │
│     └─ Only 1 replica running, no HPA/scaling policy            │
│                                                                  │
│  3. UNBOUNDED IN-MEMORY DATA STORE                              │
│     └─ Items array grows indefinitely with no cap                │
│                                                                  │
│  4. NO MEMORY ALERTS / EARLY WARNING                            │
│     └─ Alert only triggered after 10 min of health failures      │
│                                                                  │
│  5. NO REQUEST PAYLOAD SIZE LIMIT                               │
│     └─ Express accepted arbitrarily large JSON bodies            │
│                                                                  │
└──────────────────────────────────────────────────────────────────┘

3.3 Why Did Restarts Fail to Recover?

Each restart reloaded the same conditions:

Container starts → Node.js initializes (~80MB baseline)
Pending queued requests immediately hit the API
Large payloads + in-memory storage push memory past the limit
OOMKilled again within seconds
Orchestrator enters CrashLoopBackOff with increasing delays between restarts

4. Remediation Actions Taken

Action	Status
Increased container memory limit to 512MB	✅ Done (hotfix)
Restarted container with new limits	✅ Done
Confirmed service recovery and stability	✅ Done

5. Recommendations & Prevention

5.1 Application-Level Fixes

A. Limit Request Payload Size

// In src/index.js — restrict incoming JSON body size
app.use(express.json({ limit: '1mb' }));

Prevents a single large request from consuming excessive memory.

B. Cap In-Memory Data Store

// Limit the items array to prevent unbounded growth
const MAX_ITEMS = 1000;

app.post('/api/v1/items', (req, res) => {
  if (items.length >= MAX_ITEMS) {
    return res.status(429).json({
      success: false,
      error: 'Maximum item limit reached'
    });
  }
  // ... rest of handler
});

C. Set Node.js Memory Ceiling

# In Dockerfile — set explicit V8 heap limit
CMD ["node", "--max-old-space-size=384", "src/index.js"]

Ensures Node.js garbage collector runs more aggressively before hitting the container limit.

D. Use External Storage for Production

Replace the in-memory items array with a proper database (Redis, PostgreSQL, MongoDB) so application memory remains constant regardless of data volume.

5.2 Container & Infrastructure Fixes

A. Set Proper Resource Requests & Limits

# Kubernetes Deployment example
resources:
  requests:
    memory: "256Mi"    # Guaranteed minimum
    cpu: "100m"
  limits:
    memory: "512Mi"    # Hard ceiling
    cpu: "500m"

Important

Rule of thumb: Set limits.memory to at least 2x the average working set. Set requests.memory to the steady-state average.

B. Add Graceful Shutdown Handling

process.on('SIGTERM', () => {
  console.log('SIGTERM received. Shutting down gracefully...');
  server.close(() => {
    process.exit(0);
  });
});

6. Auto-Scaling Policy Design

6.1 Horizontal Pod Autoscaler (HPA) Policy

The following auto-scaling policy prevents a single container from being overwhelmed by distributing load across multiple replicas:

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: ghayma-api-hpa
  namespace: production
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: ghayma-api
  
  # Replica bounds
  minReplicas: 2          # Always run at least 2 for high availability
  maxReplicas: 10         # Cap to control costs
  
  # Scaling metrics
  metrics:
    # Scale based on MEMORY usage (primary - prevents OOMKill)
    - type: Resource
      resource:
        name: memory
        target:
          type: Utilization
          averageUtilization: 70    # Scale up when memory > 70% of limit
    
    # Scale based on CPU usage (secondary)
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 75    # Scale up when CPU > 75%
  
  # Scaling behavior (prevents flapping)
  behavior:
    scaleUp:
      stabilizationWindowSeconds: 30      # React quickly to spikes
      policies:
        - type: Pods
          value: 2                         # Add up to 2 pods at a time
          periodSeconds: 60
    scaleDown:
      stabilizationWindowSeconds: 300     # Wait 5 min before scaling down
      policies:
        - type: Pods
          value: 1                         # Remove 1 pod at a time
          periodSeconds: 120

6.2 How This Policy Prevents Duplication of the Incident

Traffic Spike Detected
        │
        ▼
┌───────────────────────┐
│ Memory usage > 70%    │──── YES ──▶ HPA adds new replicas (up to 10)
│ on existing pods?     │            Load balancer distributes traffic
└───────────────────────┘            Memory per-pod stays under limit
        │ NO                                      │
        ▼                                         ▼
  Normal operation               Spike absorbed across multiple pods
                                 No single pod reaches OOMKill threshold

6.3 Cloud Platform Auto-Scaling (Non-Kubernetes)

For managed container platforms (e.g., Ghayma Cloud, AWS ECS, Google Cloud Run):

Setting	Value	Rationale
Min instances	2	Avoid cold starts & single point of failure
Max instances	10	Cost ceiling
Scale-up trigger	Memory > 70% OR CPU > 75% OR Concurrent requests > 50	Multi-signal scaling
Scale-up cooldown	30 seconds	React quickly to spikes
Scale-down cooldown	5 minutes	Prevent flapping during variable traffic
Memory per instance	512MB	2x steady-state working set

7. Early Detection — Monitoring & Alerting Strategy

7.1 Key Metrics to Monitor

Metric	Source	Warning Threshold	Critical Threshold
Container Memory Usage	cAdvisor / Platform metrics	> 70% of limit	> 85% of limit
Container Restart Count	Kubelet / Platform events	≥ 1 restart in 5 min	≥ 3 restarts in 10 min
OOMKilled Events	Kernel / Container runtime	Any occurrence	N/A (always critical)
Response Time (p95)	Application `/metrics` endpoint	> 500ms	> 2000ms
Error Rate (5xx)	Load balancer / Application	> 1%	> 5%
Health Check Failures	Platform health probe	1 consecutive failure	3 consecutive failures
Request Queue Depth	Load balancer	> 50 pending	> 200 pending

7.2 Alerting Rules (Prometheus / Grafana Example)

# Alert: Memory approaching container limit
- alert: HighMemoryUsage
  expr: |
    (container_memory_usage_bytes{container="ghayma-api"}
    / container_spec_memory_limit_bytes{container="ghayma-api"}) > 0.70
  for: 2m
  labels:
    severity: warning
  annotations:
    summary: "Ghayma API memory usage above 70%"
    description: "Container {{ $labels.pod }} memory at {{ $value | humanizePercentage }}"

# Alert: OOMKill detected (CRITICAL — immediate page)
- alert: OOMKillDetected
  expr: |
    increase(kube_pod_container_status_restarts_total{container="ghayma-api"}[5m]) > 0
    and kube_pod_container_status_last_terminated_reason{container="ghayma-api"} == "OOMKilled"
  for: 0m
  labels:
    severity: critical
  annotations:
    summary: "OOMKill detected on Ghayma API"
    description: "Pod {{ $labels.pod }} was OOMKilled. Immediate investigation required."

# Alert: High error rate
- alert: HighErrorRate
  expr: |
    rate(http_requests_total{status=~"5.."}[5m])
    / rate(http_requests_total[5m]) > 0.05
  for: 3m
  labels:
    severity: critical
  annotations:
    summary: "Ghayma API error rate exceeds 5%"

7.3 Monitoring Architecture

┌─────────────────────────────────────────────────────────────────┐
│                    MONITORING STACK                              │
│                                                                  │
│  ┌──────────┐    ┌─────────────┐    ┌──────────────────────┐   │
│  │ Ghayma   │───▶│ Prometheus  │───▶│   Grafana Dashboard  │   │
│  │ API      │    │ (scrapes    │    │   (visualization +   │   │
│  │ /metrics │    │  /metrics)  │    │    alerting rules)   │   │
│  └──────────┘    └──────┬──────┘    └──────────────────────┘   │
│                         │                                       │
│                         ▼                                       │
│               ┌─────────────────┐                               │
│               │  Alert Manager  │                               │
│               │  (routing &     │                               │
│               │   deduplication)│                               │
│               └────────┬────────┘                               │
│                        │                                        │
│           ┌────────────┼─────────────┐                          │
│           ▼            ▼             ▼                          │
│     ┌──────────┐ ┌──────────┐ ┌───────────┐                    │
│     │  Slack   │ │ PagerDuty│ │   Email   │                    │
│     │ #alerts  │ │ on-call  │ │   team    │                    │
│     └──────────┘ └──────────┘ └───────────┘                    │
└─────────────────────────────────────────────────────────────────┘

7.4 Cloud Platform Native Tools

Cloud Platform	Monitoring Service	Key Feature for OOM Detection
AWS ECS / Fargate	CloudWatch Container Insights	`MemoryUtilization` metric + CloudWatch Alarms
Google Cloud Run	Cloud Monitoring	`container/memory/utilization` + Alert Policies
Azure Container Apps	Azure Monitor	`MemoryWorkingSet` + Metric Alerts
Kubernetes (any)	Prometheus + Grafana	`container_memory_usage_bytes` + `kube_pod_container_status_last_terminated_reason`

7.5 Recommended Grafana Dashboard Panels

Memory Usage vs Limit — Time series showing memory consumption relative to the container limit (with 70% and 85% threshold lines)
Container Restarts — Counter panel showing restart events over time
Response Time Heatmap — p50 / p95 / p99 latency distribution
Request Rate & Error Rate — Dual-axis graph (total requests + 5xx rate)
Pod/Replica Count — Shows HPA scaling activity over time

8. Lessons Learned

#	Lesson	Action Item
1	Default memory limits were set too low without load testing	Conduct load tests before production deployment
2	Single-replica deployment has no resilience	Always run ≥ 2 replicas for production services
3	Alert threshold (10 min) was too slow for a P1 outage	Reduce critical alert threshold to 2 minutes
4	In-memory data stores are dangerous without bounds	Use external databases or enforce collection size limits
5	No runbook existed for OOMKilled incidents	Create and publish an OOMKilled response runbook

9. Action Items Tracker

#	Action Item	Owner	Priority	Deadline	Status
1	Increase container memory limit to 512MB	DevOps	P0	Done	✅
2	Add `express.json({ limit: '1mb' })` payload limit	Backend	P1	+1 day	⬜
3	Implement HPA with memory-based scaling	DevOps	P1	+3 days	⬜
4	Set up Prometheus memory alerts (70% / 85%)	DevOps	P1	+3 days	⬜
5	Add OOMKilled alert rule (immediate paging)	DevOps	P0	+1 day	⬜
6	Run minimum 2 replicas in production	DevOps	P1	+1 day	⬜
7	Replace in-memory store with Redis/DB	Backend	P2	+1 week	⬜
8	Conduct load testing with realistic traffic	QA	P2	+2 weeks	⬜
9	Create OOMKilled incident runbook	DevOps	P2	+1 week	⬜
10	Build Grafana dashboard for container metrics	DevOps	P2	+1 week	⬜

Report prepared by: DevOps Team
Review date: July 26, 2026
Next review: August 2, 2026 (verify all P0/P1 items completed)

17 KiB خام اللوم التاريخ إلغاء الهروب الهروب