# Postmortem Report

## Incident Summary

On July 28, 2026, the application experienced repeated OOMKilled events, causing service interruption for approximately 45 minutes. During this period, users were unable to access the application.

---

## Timeline

- 10:00 AM - Application deployed successfully.
- 10:12 AM - Memory usage started increasing.
- 10:18 AM - First OOMKilled event occurred.
- 10:20 AM - Kubernetes restarted the container.
- 10:24 AM - Multiple OOMKilled events repeated.
- 10:45 AM - Monitoring system alerted the operations team.
- 11:00 AM - Memory limit increased and memory leak fixed.
- 11:03 AM - Service fully restored.

---

## Root Cause

The application contained a memory leak that caused continuous memory consumption. The container exceeded its memory limit, resulting in repeated OOMKilled events and continuous restarts.

---

## Recommendations

- Set appropriate memory requests and limits.
- Enable Horizontal Pod Autoscaler (HPA).
- Monitor memory usage continuously.
- Configure alerting before memory reaches critical levels.
- Perform load testing before production deployment.

---

# Auto Scaling Policy

## Objective

Prevent service interruption caused by high memory usage and repeated OOMKilled events.

## Scaling Policy

- Minimum Replicas: 2
- Maximum Replicas: 10

### Scale Out

Increase the number of containers when:

- CPU usage > 70% for 2 minutes.
- Memory usage > 75% for 2 minutes.

Increase by 2 replicas at a time.

### Scale In

Decrease the number of containers when:

- CPU usage < 40% for 10 minutes.
- Memory usage < 50% for 10 minutes.

Remove only one replica at a time.

### Additional Protection

- Configure memory requests and limits.
- Enable readiness and liveness probes.
- Prevent scheduling on unhealthy nodes.

---

# Early Detection Using Cloud Monitoring

The issue can be detected early using cloud monitoring tools by continuously monitoring application and infrastructure metrics.

## Metrics to Monitor

- Memory Usage
- CPU Usage
- Container Restarts
- OOMKilled Events
- Pod Status
- Application Response Time

## Alerts

Configure alerts when:

- Memory usage exceeds 80%.
- Container restarts more than 3 times within 5 minutes.
- OOMKilled event is detected.
- Response time exceeds 500 ms.

## Monitoring Tools

- Prometheus
- Grafana
- Kubernetes Metrics Server
- Cloud Monitoring Platform

These tools provide dashboards, real-time metrics, and alert notifications to help engineers respond before the service becomes unavailable.