Add all assessment files

q2-postmortem/postmortem-report.md

@@ -0,0 +1,123 @@
+# Incident Postmortem Report
+
+## Incident Summary
+
+On 27 July 2026, an application deployed on the Ghaymah platform experienced repeated **OOMKilled** events, resulting in approximately **45 minutes of downtime**. During this period, users were unable to access the service because the application pods kept restarting after exceeding their memory limit.
+
+---
+
+# Timeline
+
+| Time | Event |
+|------|-------|
+| 10:00 | A new application version was deployed. |
+| 10:08 | Memory usage began increasing steadily. |
+| 10:12 | The first pod was terminated with an **OOMKilled** event. |
+| 10:15 | Kubernetes automatically restarted the pod. |
+| 10:18 – 10:40 | The application entered a restart loop as multiple pods were repeatedly OOMKilled. |
+| 10:42 | The SRE team investigated pod events, logs, and resource metrics. |
+| 10:45 | Memory limits were increased temporarily, and the application's memory usage was reviewed. |
+| 10:47 | A new healthy pod started successfully. |
+| 10:55 | The service was fully restored and became available to users again. |
+
+---
+
+# Root Cause
+
+The application gradually consumed more memory than the Kubernetes memory limit that had been configured for the container. Once the limit was exceeded, Kubernetes terminated the container with an **OOMKilled** event.
+
+Although Kubernetes restarted the pod automatically, the application continued to exceed its memory limit after each restart. This created a restart loop that kept the service unavailable until the memory issue was addressed.
+
+---
+
+# Impact
+
+- Approximately **45 minutes** of service downtime.
+- Multiple pod restarts due to repeated **OOMKilled** events.
+- Users experienced failed requests and service unavailability.
+- Increased operational effort to investigate and restore the application.
+
+---
+
+# Resolution
+
+The incident was resolved by:
+
+- Investigating Kubernetes pod events and application logs.
+- Identifying excessive memory consumption.
+- Increasing the container memory limit temporarily.
+- Restarting the affected pods.
+- Verifying that the application returned to a healthy state.
+
+---
+
+# Recommendations
+
+To reduce the likelihood of similar incidents:
+
+- Configure appropriate CPU and memory requests and limits.
+- Enable Horizontal Pod Autoscaler (HPA).
+- Create alerts for high memory utilization.
+- Monitor pod restart counts and OOMKilled events.
+- Perform load testing before deploying new releases.
+- Review application memory usage regularly to identify potential memory leaks.
+
+---
+
+# Auto-Scaling Policy for Ghaymah Platform
+
+To prevent similar incidents, the platform should automatically scale the application based on resource utilization.
+
+## Horizontal Pod Autoscaler (HPA)
+
+- Minimum replicas: **2**
+- Maximum replicas: **10**
+
+### Scaling Rules
+
+- Scale out when CPU utilization exceeds **70%**.
+- Scale out when memory utilization exceeds **75%**.
+- Scale in only after resource utilization remains below **40%** for several minutes to avoid frequent scaling.
+
+This policy helps distribute incoming traffic across multiple pods before any single container reaches its memory limit.
+
+---
+
+# Early Detection Using Ghaymah Monitoring
+
+The issue can be detected before it causes downtime by continuously monitoring application and Kubernetes metrics.
+
+## Key Metrics
+
+- Container memory usage
+- Memory utilization percentage
+- CPU utilization
+- Pod restart count
+- OOMKilled events
+- API response time
+- HTTP 5xx error rate
+- Application availability
+
+## Alerting Rules
+
+Create alerts when:
+
+- Memory utilization exceeds **80%** for more than 5 minutes.
+- A pod restarts more than **3 times** within 10 minutes.
+- An **OOMKilled** event occurs.
+- API response time exceeds the defined threshold.
+- Service availability drops below the expected level.
+
+## Dashboard
+
+A monitoring dashboard should display:
+
+- CPU usage
+- Memory usage
+- Running pods
+- Pod restart count
+- Request rate
+- Response time
+- Active alerts
+
+With these metrics and alerts in place, the operations team can detect abnormal memory growth early and take corrective action before users are affected.