Complete Ghaymah SRE Technical Assessment

q2-postmortem/postmortem-report.md

@@ -0,0 +1,104 @@
+# Postmortem Report
+
+## Incident Summary
+
+On July 28, 2026, the application experienced repeated OOMKilled events, causing service interruption for approximately 45 minutes. During this period, users were unable to access the application.
+
+---
+
+## Timeline
+
+- 10:00 AM - Application deployed successfully.
+- 10:12 AM - Memory usage started increasing.
+- 10:18 AM - First OOMKilled event occurred.
+- 10:20 AM - Kubernetes restarted the container.
+- 10:24 AM - Multiple OOMKilled events repeated.
+- 10:45 AM - Monitoring system alerted the operations team.
+- 11:00 AM - Memory limit increased and memory leak fixed.
+- 11:03 AM - Service fully restored.
+
+---
+
+## Root Cause
+
+The application contained a memory leak that caused continuous memory consumption. The container exceeded its memory limit, resulting in repeated OOMKilled events and continuous restarts.
+
+---
+
+## Recommendations
+
+- Set appropriate memory requests and limits.
+- Enable Horizontal Pod Autoscaler (HPA).
+- Monitor memory usage continuously.
+- Configure alerting before memory reaches critical levels.
+- Perform load testing before production deployment.
+
+---
+
+# Auto Scaling Policy
+
+## Objective
+
+Prevent service interruption caused by high memory usage and repeated OOMKilled events.
+
+## Scaling Policy
+
+- Minimum Replicas: 2
+- Maximum Replicas: 10
+
+### Scale Out
+
+Increase the number of containers when:
+
+- CPU usage > 70% for 2 minutes.
+- Memory usage > 75% for 2 minutes.
+
+Increase by 2 replicas at a time.
+
+### Scale In
+
+Decrease the number of containers when:
+
+- CPU usage < 40% for 10 minutes.
+- Memory usage < 50% for 10 minutes.
+
+Remove only one replica at a time.
+
+### Additional Protection
+
+- Configure memory requests and limits.
+- Enable readiness and liveness probes.
+- Prevent scheduling on unhealthy nodes.
+
+---
+
+# Early Detection Using Cloud Monitoring
+
+The issue can be detected early using cloud monitoring tools by continuously monitoring application and infrastructure metrics.
+
+## Metrics to Monitor
+
+- Memory Usage
+- CPU Usage
+- Container Restarts
+- OOMKilled Events
+- Pod Status
+- Application Response Time
+
+## Alerts
+
+Configure alerts when:
+
+- Memory usage exceeds 80%.
+- Container restarts more than 3 times within 5 minutes.
+- OOMKilled event is detected.
+- Response time exceeds 500 ms.
+
+## Monitoring Tools
+
+- Prometheus
+- Grafana
+- Kubernetes Metrics Server
+- Cloud Monitoring Platform
+
+These tools provide dashboards, real-time metrics, and alert notifications to help engineers respond before the service becomes unavailable.