Q2: Answer

q2-postmortem/postmortem-report.md

@@ -0,0 +1,57 @@
+## 1. Incident Postmortem Report
+
+### Executive Summary
+
+On [27/7/2027], the main application experienced a total outage lasting for 45 minutes. System logs indicated that the downtime was caused by containers exceeding their allocated memory limits, which triggered the operating system to forcefully terminate them with an `OOMKilled` (Out Of Memory) error. The containers subsequently entered a CrashLoopBackOff state. The incident was mitigated, and service was fully restored by rolling back to the previous stable release.
+
+### Timeline
+
+- **10:00 AM:** A new application update was deployed to the production environment.
+- **10:10 AM:** Monitoring dashboards detected a sharp, abnormal spike in container memory utilization.
+- **10:20 AM:** **[Outage Start]** Container memory usage hit the 100% threshold limit. The OS kernel started terminating containers (`OOMKilled`), resulting in a complete service disruption for all users.
+- **10:45 AM:** Automated alerts were triggered, and the incident was escalated to the SRE team to investigate the continuous restart loops.
+- **10:55 AM:** Log analysis identified the root cause as a memory issue tied to the recent code deployment.
+- **11:00 AM:** An emergency rollback to the previous stable version was initiated to recover the service quickly.
+- **11:05 AM:** **[Outage End]** Containers stabilized, memory consumption returned to normal baseline levels, and the service became 100% operational.
+
+### Root Cause
+
+The primary cause of the outage was a severe memory bloat triggered by an unoptimized database query introduced in the latest release. The new code fetched a massive dataset and loaded it entirely into the container's RAM at once. As multiple user requests hit this endpoint concurrently, the memory consumption rapidly exceeded the container's hard memory limit, forcing the Kernel to kill the process (`OOMKilled`) to protect node resources.
+
+### Recommendations (Action Items)
+
+1. **Code Optimization:** The development team must implement Data Pagination or streaming for large database queries to prevent loading bulk data into memory at once.
+2. **Performance Testing:** Integrate load and stress testing into the CI/CD pipeline (Staging environment) to catch memory leaks or spikes before production deployment.
+3. **Auto-scaling & Proactive Monitoring:** (Detailed in the following sections).
+
+
+## 2. Auto-Scaling Policy Design
+
+To prevent future outages caused by heavy load and memory spikes, a **Horizontal Pod Autoscaling (HPA)** policy must be configured on the Ghaymah platform.
+
+**Policy Configuration:**
+* **Target Metric:** Memory Utilization.
+* **Scale-Out Threshold:** 70% of the allocated memory limit.
+* **Minimum Replicas:** 2 pods (to ensure basic high availability).
+* **Maximum Replicas:** 10 pods (to control resource consumption and costs).
+
+**Mechanism & Impact:**
+Instead of allowing a container's memory to reach 100% and inevitably crash, this policy acts proactively. When the platform detects that the average memory utilization across the running pods has reached the 70% threshold, it will automatically provision new container replicas (Scale Out). The incoming requests and workload will then be load-balanced across the expanded pool of containers. This distribution relieves the memory pressure on individual pods, ensuring no single container hits the critical limit that triggers an `OOMKilled` event.
+
+
+## 3. Proactive Monitoring & Early Detection Strategy
+
+To ensure anomalies are detected and mitigated before they impact end-users, we must shift from reactive troubleshooting to proactive observability. The following alerting strategy will be configured within the platform's monitoring tools (e.g., Prometheus/Alertmanager):
+
+**1. Resource Utilization Alerts:**
+*   **Warning Alert (Yellow):** 
+    *   **Condition:** Container memory utilization exceeds **75%** for more than 3 consecutive minutes.
+    *   **Action:** Send an automated notification to the engineering team's **Slack** channel. This allows developers to investigate potential memory leaks early without waking up the on-call engineer.
+*   **Critical Alert (Red):** 
+    *   **Condition:** Container memory utilization reaches **85%**.
+    *   **Action:** Trigger a **PagerDuty** incident to automatically call the on-call SRE. This requires immediate manual intervention to scale the system or rollback before an `OOMKilled` event occurs.
+
+**2. Container Health & Churn Monitoring:**
+*   **CrashLoop Alert:** 
+    *   **Condition:** The `kube_pod_container_status_restarts_total` metric (RestartCount) increases by more than **1** within a **5-minute** window.
+    *   **Action:** Send a high-priority alarm indicating a failing application or a potential CrashLoopBackOff state, ensuring the team is aware of instability even if the system is currently auto-recovering.