feat: complete SRE exam deliverables for Ghaymah & Qabilah platform submission

common-mortakaz/integration-2.md

@@ -0,0 +1,58 @@
+# Mortakaz Integration Spec 2 — Automated Incident Management & Observability Pipeline
+
+## Executive Overview
+
+This specification details the event-driven incident integration between **Ghaymah Cloud Monitoring** and **Mortakaz Incident Response Platform**. When container outages (such as `OOMKilled` events) or threshold breaches occur on Ghaymah, automated webhook alerts trigger incident workflows on Mortakaz.
+
+---
+
+## 1. Event-Driven Incident Lifecycle
+
+```
+[ Ghaymah Cloud Pod ] ──(OOMKilled Event)──> [ Ghaymah Alertmanager ]
+                                                     │
+                                                     ▼ Webhook
+                                          [ Mortakaz Webhook Listener ]
+                                                     │
+                                                     ▼
+                                          [ Auto-Create Incident Ticket ]
+                                                     │
+                                                     ▼
+                                          [ Trigger HPA / Auto-Remediation ]
+```
+
+---
+
+## 2. Mortakaz Webhook Event Schema
+
+When Ghaymah detects a critical alert (e.g. memory saturation > 88% or pod crash), it posts the following JSON payload to Mortakaz:
+
+```json
+{
+  "event_id": "evt_98410294812",
+  "event_type": "CONTAINER_OOM_KILLED",
+  "severity": "CRITICAL",
+  "qabilah_user": "marwanabdelmoneim",
+  "email": "marwantamermo@gmail.com",
+  "affected_resource": {
+    "platform": "ghaymah.systems",
+    "namespace": "production",
+    "deployment": "ghaymah-api-deployment",
+    "pod_id": "ghaymah-api-7f8d9-x4k21",
+    "exit_code": 137,
+    "memory_limit_bytes": 536870912,
+    "memory_used_bytes": 536870912
+  },
+  "remediation_action_triggered": "AUTOMATIC_HPA_SCALE_UP",
+  "timestamp": "2026-07-27T14:19:00Z"
+}
+```
+
+---
+
+## 3. Automated Remediation Workflow
+
+1. **Mortakaz Trigger**: Receives `CONTAINER_OOM_KILLED` event.
+2. **Auto Scale Execution**: Invokes Ghaymah API endpoint `/v1/deployments/ghaymah-api-deployment/scale` to increase replica count from $N$ to $N+8$.
+3. **Paging & Notification**: Pages the SRE on-call engineer via Mortakaz Mobile App and Slack `#sre-incidents` channel.
+4. **Postmortem Auto-Drafting**: Generates initial incident timeline data for postmortem analysis.