# Incident Postmortem: Core Backend API Service Interruption

> **Incident Date:** July 27, 2026  
> **Affected Service:** Core .NET Backend API (Ghaymah Platform)  
> **Severity:** SEV-1 (Critical)  
> **Status:** Resolved

---

# Table of Contents

- [Incident Overview](#incident-overview)
- [Incident Metadata](#incident-metadata)
- [Executive Summary](#executive-summary)
- [Incident Timeline](#incident-timeline)
- [Root Cause Analysis](#root-cause-analysis)
- [Proposed Auto-Scaling Architecture](#proposed-auto-scaling-architecture)
- [Horizontal Pod Autoscaler Configuration](#horizontal-pod-autoscaler-configuration)
- [Observability & Early Detection](#observability--early-detection)
- [Action Items](#action-items)
- [Lessons Learned](#lessons-learned)

---

# Incident Overview

On **July 27, 2026**, the **Core .NET Backend API** experienced a complete service outage lasting **45 minutes**.

The outage was caused by repeated **OOMKilled (Exit Code 137)** events after the application exhausted its available memory during a sudden traffic spike.

The platform entered a continuous **CrashLoop** state, resulting in **100% API failure** until memory resources were increased and the containers were restarted.

---

# Incident Metadata

| Category | Details |
|-----------|---------|
| **Affected Service** | Core .NET Backend API |
| **Platform** | Ghaymah |
| **Date** | 2026-07-27 |
| **Downtime** | 45 Minutes |
| **Time** | 14:00 – 14:45 EEST |
| **Severity** | SEV-1 (Critical) |
| **Customer Impact** | 100% API transaction failures |
| **Observed Errors** | 502 Bad Gateway |

---

# Executive Summary

A sudden **400% increase in traffic** caused the backend API to consume memory rapidly.

The application maintained an **unbounded in-memory cache**, storing increasingly large datasets without expiration or eviction.

Once the container reached its configured memory limit, the Linux kernel terminated the process (**OOMKilled - Exit Code 137**) to protect node stability.

The orchestration platform continuously restarted the containers, causing a **CrashLoopBackOff** cycle and preventing the service from recovering automatically.

Service was restored after:

- Increasing the container memory limit
- Restarting the backend pods
- Verifying successful application startup

Future mitigation requires:

- Memory-based auto-scaling
- Cache eviction policies
- API pagination
- Improved monitoring and alerting

---

# Incident Timeline

| Time | Event |
|------|-------|
| **13:50** | Traffic increased by approximately **400%** due to an unexpected external campaign. |
| **13:58** | Container memory utilization exceeded **95%** of allocated memory. |
| **14:00** | First container terminated with **OOMKilled (Exit Code 137)**. Service degradation begins. |
| **14:05 – 14:30** | Containers repeatedly restarted by the orchestrator, resulting in a CrashLoop and complete outage. |
| **14:30** | On-call infrastructure engineer identified repeated OOMKilled events from platform logs. |
| **14:35** | Memory limit increased from **512Mi** to **2Gi** and backend pods restarted manually. |
| **14:45** | Service stabilized and API responses returned **HTTP 200 OK**. Incident closed. |

---

# Root Cause Analysis

## Direct Cause

The Linux kernel terminated the backend process because the application attempted to allocate more memory than the container's configured memory limit.

```
Exit Code: 137
Reason: OOMKilled
```

---

## Underlying Causes

### Unbounded In-Memory Cache

The application stored data inside a local in-memory dictionary that had:

- No Time-To-Live (TTL)
- No maximum cache size
- No eviction policy

Large requests continuously expanded the cache until the process exhausted available memory.

---

### Missing API Pagination

Several endpoints returned very large datasets.

Without pagination:

- Large payloads were cached
- Memory usage increased rapidly
- Garbage collection became inefficient

---

### Lack of Horizontal Auto-Scaling

The deployment relied on static memory limits and a fixed number of replicas.

As traffic increased:

- No new replicas were created
- Existing containers absorbed all incoming traffic
- Memory utilization reached critical levels

---

# Proposed Auto-Scaling Architecture

To prevent similar incidents, backend workloads should scale automatically based on memory utilization.

| Parameter | Recommended Value |
|-----------|-------------------|
| **Scaling Metric** | Average Memory Utilization |
| **Target Utilization** | 75% |
| **Minimum Replicas** | 3 |
| **Maximum Replicas** | 12 |
| **Scale-Up Policy** | Add up to 4 replicas immediately |
| **Scale-Down Policy** | Wait 5 minutes after memory falls below 40% |

---

# Horizontal Pod Autoscaler Configuration

```yaml
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: backend-api-scaler

spec:
  minReplicas: 3
  maxReplicas: 12

  metrics:
    - type: Resource
      resource:
        name: memory
        target:
          type: Utilization
          averageUtilization: 75
```

---

# Observability & Early Detection

To move from reactive troubleshooting to proactive monitoring, the following observability improvements should be implemented.

## Alerting Rules

### Memory Utilization Alert

Trigger notification when:

- Memory utilization exceeds **70%**
- Sustained for **2 consecutive minutes**

Notification targets:

- Email
- Slack
- Webhook

---

### CrashLoop Detection

Create a high-priority incident whenever:

- Container restart count exceeds **3**
- Within a **10-minute** window

---

## Dashboard Metrics

Create a dedicated monitoring dashboard showing:

- Container memory usage
- Container memory limits
- Application heap size
- Garbage Collection duration
- API request rate
- Current replica count
- Container restart count
- CPU utilization
- Response latency
- Error rate (4xx / 5xx)

---

# Action Items

| Task | Owner | Priority | Status |
|------|-------|----------|--------|
| Implement cache eviction (LRU + TTL) | Development Team | Critical | In Progress |
| Enforce API pagination | Development Team | Critical | In Progress |
| Deploy memory-based Horizontal Pod Autoscaler | DevOps Team | High | Not Started |
| Create synthetic load tests (5× traffic) | QA Team | Medium | Not Started |
| Update on-call operational runbooks | Operations Team | Low | Completed |

---

# Lessons Learned

The incident highlighted several architectural improvements required to increase platform resilience.

## Infrastructure

- Configure Horizontal Pod Autoscaler (HPA)
- Define resource requests and limits carefully
- Monitor memory consumption continuously

## Application

- Implement cache eviction (LRU)
- Apply cache expiration (TTL)
- Enforce pagination on all large API endpoints
- Optimize memory allocation patterns

## Operations

- Improve proactive alerting
- Expand observability dashboards
- Regularly execute load and stress testing
- Update incident response runbooks

---

# Resolution Summary

| Item | Result |
|------|--------|
| **Root Cause** | Unbounded in-memory cache caused container OOM |
| **Immediate Fix** | Increased memory limit (512Mi → 2Gi) and restarted pods |
| **Long-Term Fixes** | HPA, cache eviction, pagination, monitoring improvements |
| **Incident Status** | ✅ Resolved |
```