Initial Ghaymah tasks setup

task2-postmortem/postmortem.md

@@ -0,0 +1,121 @@
+# Incident Postmortem Report
+## Application Outage Due to Repeated OOMKilled Events
+
+## 1. Incident Overview
+Incident ID: INC-2026-001
+Severity: SEV-2
+Duration: 45 minutes
+Service: Ghaymah Hosted Application
+Status: Resolved
+
+The application experienced repeated downtime caused by Kubernetes OOMKilled events.
+The container exceeded its allocated memory limit, causing Kubernetes to terminate
+and restart the container repeatedly.
+
+
+## 2. Executive Summary
+On July 28, 2026, the application experienced intermittent availability issues
+for approximately 45 minutes.
+
+The root cause was excessive memory consumption inside the application container.
+Once the container exceeded its configured memory limit, Kubernetes terminated
+the container with an OOMKilled event.
+
+The service recovered after adjusting resource configuration and implementing
+a better scaling strategy.
+
+## 3. Impact
+During the incident:
+- Users experienced 502/503 errors.
+- API requests failed intermittently.
+- Application availability decreased for 45 minutes.
+- No data loss occurred because the application was stateless.
+
+
+## 4. Detection
+The incident was detected through:
+- Kubernetes container restart events.
+- OOMKilled status reported by the platform.
+- Increased application response latency.
+- Monitoring alerts triggered by high memory utilization.
+
+## 5. Timeline
+| Time | Event |
+|---|---|
+| 10:00 | Traffic increased above normal level |
+| 10:02 | Container memory usage exceeded 90% |
+| 10:04 | Kubernetes terminated container (OOMKilled) |
+| 10:05 | New container instance started |
+| 10:10 | Memory increased again and container crashed |
+| 10:45 | Service stabilized after mitigation |
+
+## 6. Root Cause Analysis
+
+### Immediate Cause
+The container exceeded its configured memory limit.
+
+### Root Cause
+The application was running with insufficient memory resources
+and no horizontal scaling mechanism.
+Possible contributing factors:
+- Memory leak inside the application.
+- Missing memory-based autoscaling.
+- No early warning alerts.
+- Single container handling all traffic.
+
+## 7. Resolution
+The incident was resolved by:
+
+- Increasing container memory allocation.
+- Restarting unhealthy workloads.
+- Reviewing application memory usage.
+- Preparing an autoscaling policy.
+
+## 8. Preventive Actions
+| Action | Priority |
+|---|---|
+| Enable Horizontal Pod Autoscaler | High |
+| Add memory utilization alerts | High |
+| Perform memory profiling | Medium |
+| Improve health checks | Medium |
+| Add resource limits based on metrics | High |
+
+
+autoscaling:
+  minReplicas: 2
+  maxReplicas: 8
+
+  metrics:
+    memory:
+      targetUtilization: 70
+
+    cpu:
+      targetUtilization: 75
+
+  behavior:
+    scaleUp:
+      stabilizationWindowSeconds: 60
+
+    scaleDown:
+      stabilizationWindowSeconds: 300
+
+resources:
+  requests:
+    memory: 256Mi
+    cpu: 500m
+
+  limits:
+    memory: 512Mi
+    cpu: 1000m
+
+
+## Monitoring Strategy
+The following monitoring improvements will be implemented:
+- Memory usage dashboard.
+- Alert when memory exceeds 75%.
+- Critical alert when memory exceeds 90%.
+- Monitor container restart count.
+- Track OOMKilled events.
+- Synthetic health checks.
+
+