# Incident Postmortem Report

## Incident Summary

On 27 July 2026, an application deployed on the Ghaymah platform experienced repeated **OOMKilled** events, resulting in approximately **45 minutes of downtime**. During this period, users were unable to access the service because the application pods kept restarting after exceeding their memory limit.

---

# Timeline

| Time | Event |
|------|-------|
| 10:00 | A new application version was deployed. |
| 10:08 | Memory usage began increasing steadily. |
| 10:12 | The first pod was terminated with an **OOMKilled** event. |
| 10:15 | Kubernetes automatically restarted the pod. |
| 10:18 – 10:40 | The application entered a restart loop as multiple pods were repeatedly OOMKilled. |
| 10:42 | The SRE team investigated pod events, logs, and resource metrics. |
| 10:45 | Memory limits were increased temporarily, and the application's memory usage was reviewed. |
| 10:47 | A new healthy pod started successfully. |
| 10:55 | The service was fully restored and became available to users again. |

---

# Root Cause

The application gradually consumed more memory than the Kubernetes memory limit that had been configured for the container. Once the limit was exceeded, Kubernetes terminated the container with an **OOMKilled** event.

Although Kubernetes restarted the pod automatically, the application continued to exceed its memory limit after each restart. This created a restart loop that kept the service unavailable until the memory issue was addressed.

---

# Impact

- Approximately **45 minutes** of service downtime.
- Multiple pod restarts due to repeated **OOMKilled** events.
- Users experienced failed requests and service unavailability.
- Increased operational effort to investigate and restore the application.

---

# Resolution

The incident was resolved by:

- Investigating Kubernetes pod events and application logs.
- Identifying excessive memory consumption.
- Increasing the container memory limit temporarily.
- Restarting the affected pods.
- Verifying that the application returned to a healthy state.

---

# Recommendations

To reduce the likelihood of similar incidents:

- Configure appropriate CPU and memory requests and limits.
- Enable Horizontal Pod Autoscaler (HPA).
- Create alerts for high memory utilization.
- Monitor pod restart counts and OOMKilled events.
- Perform load testing before deploying new releases.
- Review application memory usage regularly to identify potential memory leaks.

---

# Auto-Scaling Policy for Ghaymah Platform

To prevent similar incidents, the platform should automatically scale the application based on resource utilization.

## Horizontal Pod Autoscaler (HPA)

- Minimum replicas: **2**
- Maximum replicas: **10**

### Scaling Rules

- Scale out when CPU utilization exceeds **70%**.
- Scale out when memory utilization exceeds **75%**.
- Scale in only after resource utilization remains below **40%** for several minutes to avoid frequent scaling.

This policy helps distribute incoming traffic across multiple pods before any single container reaches its memory limit.

---

# Early Detection Using Ghaymah Monitoring

The issue can be detected before it causes downtime by continuously monitoring application and Kubernetes metrics.

## Key Metrics

- Container memory usage
- Memory utilization percentage
- CPU utilization
- Pod restart count
- OOMKilled events
- API response time
- HTTP 5xx error rate
- Application availability

## Alerting Rules

Create alerts when:

- Memory utilization exceeds **80%** for more than 5 minutes.
- A pod restarts more than **3 times** within 10 minutes.
- An **OOMKilled** event occurs.
- API response time exceeds the defined threshold.
- Service availability drops below the expected level.

## Dashboard

A monitoring dashboard should display:

- CPU usage
- Memory usage
- Running pods
- Pod restart count
- Request rate
- Response time
- Active alerts

With these metrics and alerts in place, the operations team can detect abnormal memory growth early and take corrective action before users are affected.
