# Incident Postmortem Report
## Application Outage Due to Repeated OOMKilled Events

## 1. Incident Overview
Incident ID: INC-2026-001
Severity: SEV-2
Duration: 45 minutes
Service: Ghaymah Hosted Application
Status: Resolved

The application experienced repeated downtime caused by Kubernetes OOMKilled events.
The container exceeded its allocated memory limit, causing Kubernetes to terminate
and restart the container repeatedly.


## 2. Executive Summary
On July 28, 2026, the application experienced intermittent availability issues
for approximately 45 minutes.

The root cause was excessive memory consumption inside the application container.
Once the container exceeded its configured memory limit, Kubernetes terminated
the container with an OOMKilled event.

The service recovered after adjusting resource configuration and implementing
a better scaling strategy.

## 3. Impact
During the incident:
- Users experienced 502/503 errors.
- API requests failed intermittently.
- Application availability decreased for 45 minutes.
- No data loss occurred because the application was stateless.


## 4. Detection
The incident was detected through:
- Kubernetes container restart events.
- OOMKilled status reported by the platform.
- Increased application response latency.
- Monitoring alerts triggered by high memory utilization.

## 5. Timeline
| Time | Event |
|---|---|
| 10:00 | Traffic increased above normal level |
| 10:02 | Container memory usage exceeded 90% |
| 10:04 | Kubernetes terminated container (OOMKilled) |
| 10:05 | New container instance started |
| 10:10 | Memory increased again and container crashed |
| 10:45 | Service stabilized after mitigation |

## 6. Root Cause Analysis

### Immediate Cause
The container exceeded its configured memory limit.

### Root Cause
The application was running with insufficient memory resources
and no horizontal scaling mechanism.
Possible contributing factors:
- Memory leak inside the application.
- Missing memory-based autoscaling.
- No early warning alerts.
- Single container handling all traffic.

## 7. Resolution
The incident was resolved by:

- Increasing container memory allocation.
- Restarting unhealthy workloads.
- Reviewing application memory usage.
- Preparing an autoscaling policy.

## 8. Preventive Actions
| Action | Priority |
|---|---|
| Enable Horizontal Pod Autoscaler | High |
| Add memory utilization alerts | High |
| Perform memory profiling | Medium |
| Improve health checks | Medium |
| Add resource limits based on metrics | High |


autoscaling:
  minReplicas: 2
  maxReplicas: 8

  metrics:
    memory:
      targetUtilization: 70

    cpu:
      targetUtilization: 75

  behavior:
    scaleUp:
      stabilizationWindowSeconds: 60

    scaleDown:
      stabilizationWindowSeconds: 300

resources:
  requests:
    memory: 256Mi
    cpu: 500m

  limits:
    memory: 512Mi
    cpu: 1000m


## Monitoring Strategy
The following monitoring improvements will be implemented:
- Memory usage dashboard.
- Alert when memory exceeds 75%.
- Critical alert when memory exceeds 90%.
- Monitor container restart count.
- Track OOMKilled events.
- Synthetic health checks.