Postmortem Report

Incident Summary

Incident Title: Repeated OOMKilled Causing Application Downtime

Date: 2026-07-28

Duration: 45 minutes

Severity: High (SEV-1)

Impact: The application became unavailable for 45 minutes because the container was repeatedly terminated with the OOMKilled status. During the outage, users were unable to access the application, resulting in service disruption.

Timeline

Time	Event
14:00	Increased traffic caused memory usage to rise.
14:05	Container exceeded its memory limit and was terminated (OOMKilled).
14:06	Kubernetes restarted the container automatically.
14:10 - 14:40	Continuous restart loop occurred due to repeated OOMKilled events.
14:25	Monitoring system generated high memory usage alerts.
14:35	SRE team began investigating pod events and resource metrics.
14:45	Memory limit was increased and the deployment was restarted.
14:45	Application recovered successfully.

Root Cause Analysis

Primary Root Cause

The application exceeded the configured memory limit. When the container reached the limit, the Linux Out Of Memory (OOM) Killer terminated the process.

Contributing Factors

Memory limits were configured too low.
No Horizontal Pod Autoscaler (HPA) was configured.
Monitoring alerts were triggered after the application became unstable.
No load testing had been performed before production deployment.

Resolution

The following actions restored the service:

Increased container memory limits.
Restarted the deployment.
Verified application health using /health.
Confirmed stable memory usage.

Preventive Actions

Immediate

Increase memory requests and limits.
Add memory usage alerts.
Perform load testing before deployment.

Long-term

Configure Horizontal Pod Autoscaler (HPA).
Implement automatic scaling based on CPU and Memory.
Enable centralized logging.
Create SLOs and alerting rules.
Review resource requests for every deployment.

Auto-Scaling Policy

To prevent similar incidents:

Minimum Replicas: 2
Maximum Replicas: 10
Target CPU Utilization: 70%
Target Memory Utilization: 75%
Scale Up immediately when thresholds are exceeded.
Scale Down gradually after 5 minutes of stable utilization.
Configure PodDisruptionBudget to maintain availability.

Early Detection Using Ghaymah Monitoring

The issue can be detected early by monitoring:

Container Memory Usage
Pod Restart Count
OOMKilled Events
Container Health Checks
Response Time
Error Rate (5xx)
Request Rate
Application Availability

Recommended alerts:

Memory Usage > 80%
Restart Count > 3 within 5 minutes
Health Check Failure
Response Time > 500 ms
Availability below 99%

Lessons Learned

Configure appropriate resource limits.
Enable proactive monitoring and alerting.
Test the application under expected production load.
Implement autoscaling before deploying production workloads.
Continuously review resource utilization trends.

3.1 KiB خام اللوم التاريخ