diff --git a/q2-postmortem/postmortem-report.md b/q2-postmortem/postmortem-report.md new file mode 100644 index 0000000..d03dc5f --- /dev/null +++ b/q2-postmortem/postmortem-report.md @@ -0,0 +1,173 @@ +# Q2 - Postmortem Report + +## Incident Summary + +**Incident:** Repeated OOMKilled causing service outage + +**Date:** (Exam Scenario) + +**Duration:** 45 minutes + +**Severity:** High (SEV-1) + +**Impact:** +- API became unavailable. +- Users received HTTP 5xx errors. +- Service uptime was affected. +- Customer requests could not be processed during the outage. + +--- + +# Timeline + +| Time | Event | +|------|-------| +| 10:00 | Increased memory usage observed. | +| 10:05 | First container terminated with OOMKilled. | +| 10:06 | Kubernetes restarted the container automatically. | +| 10:10 | Memory usage increased again causing another OOMKilled. | +| 10:15 | Continuous restart loop (CrashLoopBackOff). | +| 10:20 | Monitoring system generated alerts. | +| 10:30 | SRE team started investigation. | +| 10:40 | Memory limit increased and memory leak identified. | +| 10:45 | New container deployed successfully and service restored. | + +--- + +# Root Cause Analysis + +The application suffered from excessive memory consumption caused by a memory leak under increased traffic. + +The container memory limit was too low to handle the workload. + +When memory usage exceeded the configured limit, Kubernetes terminated the container with an **OOMKilled** event. + +Automatic restarts repeatedly created a CrashLoopBackOff situation, extending the outage. + +--- + +# Contributing Factors + +- No Horizontal Pod Autoscaler (HPA). +- Memory limits configured too aggressively. +- No early memory usage alerts. +- Memory leak was not detected during testing. +- Lack of load testing before deployment. + +--- + +# Immediate Actions Taken + +- Increased memory limits. +- Restarted affected containers. +- Fixed memory leak. +- Verified application health. +- Monitored memory utilization after recovery. + +--- + +# Preventive Recommendations + +- Enable Horizontal Pod Autoscaler (HPA). +- Configure Vertical Pod Autoscaler (VPA) if appropriate. +- Define proper memory requests and limits. +- Add Prometheus memory monitoring. +- Configure Alertmanager notifications. +- Perform stress testing before production deployment. +- Review application memory usage regularly. + +--- + +# Auto-Scaling Policy + +To prevent similar incidents on Ghaymah: + +### Horizontal Pod Autoscaler + +Minimum Replicas: 2 + +Maximum Replicas: 10 + +Scale Out Conditions + +- Memory usage > 70% +- CPU usage > 70% +- Request rate exceeds expected capacity + +Scale In Conditions + +- Memory usage < 40% +- CPU usage < 40% +- Stable traffic for at least 10 minutes + +Cooldown Period + +- Scale Out: 60 seconds + +- Scale In: 300 seconds + +Benefits + +- Prevents memory exhaustion. +- Distributes incoming traffic. +- Improves availability. +- Reduces restart frequency. + +--- + +# Early Detection Using Ghaymah Monitoring + +The incident could have been detected earlier using monitoring tools. + +## Metrics + +- Container Memory Usage +- Memory Limit Percentage +- Container Restart Count +- OOMKilled Events +- CPU Utilization +- Request Rate +- Response Time + +## Alerts + +Critical Alert + +- Memory Usage > 85% +- More than 3 restarts within 5 minutes +- OOMKilled event detected + +Warning Alert + +- Memory Usage > 70% +- Rapid increase in memory consumption + +## Dashboards + +Recommended dashboard should display: + +- Memory Usage +- CPU Usage +- Pod Status +- Restart Count +- Response Time +- Error Rate +- Availability + +--- + +# Lessons Learned + +- Memory monitoring must be proactive. +- Autoscaling should be enabled in production. +- Load testing should validate memory consumption. +- Alerting should notify engineers before service failure. +- Capacity planning should be reviewed regularly. + +--- + +# Status + +✅ Incident resolved + +No recurring issues observed after implementing corrective actions.