adding task files

q2-postmortem/postmortem-report.md

@@ -0,0 +1,120 @@
+# Postmortem Report
+
+## Incident Summary
+
+**Incident Title:** Repeated OOMKilled Causing Application Downtime
+
+**Date:** 2026-07-28
+
+**Duration:** 45 minutes
+
+**Severity:** High (SEV-1)
+
+**Impact:**
+The application became unavailable for 45 minutes because the container was repeatedly terminated with the `OOMKilled` status. During the outage, users were unable to access the application, resulting in service disruption.
+
+---
+
+# Timeline
+
+| Time | Event |
+|------|-------|
+| 14:00 | Increased traffic caused memory usage to rise. |
+| 14:05 | Container exceeded its memory limit and was terminated (OOMKilled). |
+| 14:06 | Kubernetes restarted the container automatically. |
+| 14:10 - 14:40 | Continuous restart loop occurred due to repeated OOMKilled events. |
+| 14:25 | Monitoring system generated high memory usage alerts. |
+| 14:35 | SRE team began investigating pod events and resource metrics. |
+| 14:45 | Memory limit was increased and the deployment was restarted. |
+| 14:45 | Application recovered successfully. |
+
+---
+
+# Root Cause Analysis
+
+### Primary Root Cause
+
+The application exceeded the configured memory limit. When the container reached the limit, the Linux Out Of Memory (OOM) Killer terminated the process.
+
+### Contributing Factors
+
+- Memory limits were configured too low.
+- No Horizontal Pod Autoscaler (HPA) was configured.
+- Monitoring alerts were triggered after the application became unstable.
+- No load testing had been performed before production deployment.
+
+---
+
+# Resolution
+
+The following actions restored the service:
+
+- Increased container memory limits.
+- Restarted the deployment.
+- Verified application health using `/health`.
+- Confirmed stable memory usage.
+
+---
+
+# Preventive Actions
+
+### Immediate
+
+- Increase memory requests and limits.
+- Add memory usage alerts.
+- Perform load testing before deployment.
+
+### Long-term
+
+- Configure Horizontal Pod Autoscaler (HPA).
+- Implement automatic scaling based on CPU and Memory.
+- Enable centralized logging.
+- Create SLOs and alerting rules.
+- Review resource requests for every deployment.
+
+---
+
+# Auto-Scaling Policy
+
+To prevent similar incidents:
+
+- Minimum Replicas: **2**
+- Maximum Replicas: **10**
+- Target CPU Utilization: **70%**
+- Target Memory Utilization: **75%**
+- Scale Up immediately when thresholds are exceeded.
+- Scale Down gradually after 5 minutes of stable utilization.
+- Configure PodDisruptionBudget to maintain availability.
+
+---
+
+# Early Detection Using Ghaymah Monitoring
+
+The issue can be detected early by monitoring:
+
+- Container Memory Usage
+- Pod Restart Count
+- OOMKilled Events
+- Container Health Checks
+- Response Time
+- Error Rate (5xx)
+- Request Rate
+- Application Availability
+
+Recommended alerts:
+
+- Memory Usage > 80%
+- Restart Count > 3 within 5 minutes
+- Health Check Failure
+- Response Time > 500 ms
+- Availability below 99%
+
+---
+
+# Lessons Learned
+
+- Configure appropriate resource limits.
+- Enable proactive monitoring and alerting.
+- Test the application under expected production load.
+- Implement autoscaling before deploying production workloads.
+- Continuously review resource utilization trends.