Add postmortem

q2-postmortem/Auto-Scaling.md

@@ -0,0 +1,68 @@
+# Ghaymah Auto-Scaling Policy
+
+## Goal
+
+Design an auto-scaling policy for the Ghaymah platform that reduces the impact of memory-related incidents (like the OOMKilled outage in the postmortem), while making clear that auto-scaling alone does not fix a memory leak — it only limits the damage.
+
+---
+
+## Important Note
+
+The original incident was caused by a **memory leak**, not by a real traffic increase. Auto-scaling can't fix a leak — a new pod created by auto-scaling will eventually leak too. So this policy is a **safety net**, not a replacement for fixing the leak itself.
+
+---
+
+## 1. Right-Size Requests and Limits First
+
+Before any auto-scaling works well, the memory `requests` and `limits` need to be based on real usage instead of a guess. Otherwise the autoscaler is scaling based on wrong numbers.
+
+---
+
+## 2. Horizontal Pod Autoscaler (HPA) — based on memory
+
+Scale the number of pods when average memory usage crosses a threshold, so no single pod gets close to its limit as fast.
+
+```yaml
+apiVersion: autoscaling/v2
+kind: HorizontalPodAutoscaler
+metadata:
+  name: ghaymah-app-hpa
+spec:
+  scaleTargetRef:
+    apiVersion: apps/v1
+    kind: Deployment
+    name: ghaymah-app
+  minReplicas: 2
+  maxReplicas: 6
+  metrics:
+  - type: Resource
+    resource:
+      name: memory
+      target:
+        type: Utilization
+        averageUtilization: 70
+```
+
+---
+
+## 3. Cluster Autoscaler
+
+If HPA creates more pods than the current nodes can fit, the Cluster Autoscaler adds more nodes automatically so pods aren't stuck pending.
+
+---
+
+## 4. Alerting Before Scaling Limits Are Hit
+
+Add an alert when memory usage is trending up (not just when it hits the limit), so the team can catch a leak early instead of waiting for auto-scaling or a crash.
+
+---
+
+## 5. Restart / Readiness Behavior
+
+Make sure the app has proper readiness probes, so traffic isn't sent to a pod that just restarted and isn't ready yet — this reduces user-facing errors during restart loops.
+
+---
+
+## Summary
+
+This policy helps absorb real load spikes and buys extra time before hitting memory limits, which reduces how often OOMKilled events happen. But it does **not** replace fixing the actual memory leak — that's still the real fix.

q2-postmortem/Monitoring.md

@@ -0,0 +1,29 @@
+# Early Detection of the Memory Leak Using Ghaymah's Monitoring Tools
+
+## Idea
+
+The problem should ideally be caught before memory hits the limit and the pod crashes, not after. This means watching how memory usage changes over time, not just checking if it already reached a critical point.
+
+---
+
+## 1. Watch Memory Usage Over Time (Prometheus)
+
+Instead of only alerting when memory is almost at the limit, we can track memory usage per pod using Prometheus and set an alert for when it keeps increasing steadily over several minutes without going back down. That's usually a sign of a leak, and it can be noticed while memory is still at a safe level.
+
+---
+
+## 2. Use Grafana to Visualize It
+
+A simple Grafana graph of memory usage per pod makes a leak easy to notice — instead of the usage going up and down normally, you'd see a line that keeps climbing until it crashes and restarts, then climbs again. That pattern is a clear sign something's wrong.
+
+---
+
+## 3. Track Restart Count
+
+Kubernetes also exposes how many times a pod restarted. If a pod is restarting a lot in a short time, that alone is a signal to check what's going on, even before looking at memory numbers specifically.
+
+---
+
+## Summary
+
+Instead of just alerting "memory is almost full" (which is too late), the goal is to alert on the memory **trend** going up steadily and on repeated restarts. Using Prometheus + Grafana for this would have shown the problem building up before the first crash actually happened.

q2-postmortem/Postmortem.md

@@ -0,0 +1,57 @@
+# Ghaymah Postmortem Report
+
+## Incident Summary
+
+On July 27, 2026, the Ghaymah application experienced a service outage lasting approximately 45 minutes. The outage was caused by repeated OOMKilled events: the application's memory usage kept increasing until it hit the container's memory limit, causing the pod to crash and restart. Since the memory issue wasn't fixed, this kept happening in a loop until the memory limit was increased and the service stabilized.
+
+**Severity:** SEV-2
+
+---
+
+## Impact
+
+- The application was unavailable / responded with errors on and off for about 45 minutes.
+- Requests failed whenever the pod was restarting.
+- No data was lost.
+- Service went back to normal after the memory limit was increased.
+
+---
+
+## Timeline
+
+| Time | Event |
+|------|-------|
+| 12:00 | Application running normally. |
+| 12:05 | Memory usage started increasing. |
+| 12:06 | First OOMKilled event, pod restarted automatically. |
+| 12:08 | Monitoring alert triggered (high memory / restart). |
+| 12:10 – 12:40 | Memory kept climbing after each restart, causing repeated OOMKilled events. |
+| 12:42 | Root cause identified (memory leak + low memory limit). |
+| 12:45 | Memory limit increased and deployment restarted. |
+| 12:50 | Service stable again. |
+
+---
+
+## Root Cause
+
+The main cause was a memory leak in the application, which made memory usage grow continuously instead of staying stable. Because the memory limit set on the container was low compared to this growing usage, the limit was hit quickly and Kubernetes kept killing and restarting the pod (OOMKilled), which is why the outage lasted for several restart cycles instead of just one crash.
+
+So: the leak was the actual bug, and the low memory limit is why it turned into a repeated 45-minute outage instead of a single failure.
+
+---
+
+## Recommendations
+
+- Fix the memory leak in the application.
+- Set a more realistic memory request/limit based on real usage, not a guess.
+- Add an alert for high memory usage (e.g. 80%) before it reaches the limit.
+- Consider auto-scaling so the app can handle load spikes without hitting the limit.
+- Test the app under load before deploying, to catch this kind of issue earlier.
+
+---
+
+## Lessons Learned
+
+- A memory leak can cause a much bigger outage than expected if the memory limit isn't sized correctly.
+- Auto-restart helps keep the app "up," but it hides the real problem instead of fixing it.
+- Alerting on memory usage trends (not just on crashes) would have caught this earlier.