# Mortakaz Integration Spec 2 — Automated Incident Management & Observability Pipeline

## Executive Overview

This specification details the event-driven incident integration between **Ghaymah Cloud Monitoring** and **Mortakaz Incident Response Platform**. When container outages (such as `OOMKilled` events) or threshold breaches occur on Ghaymah, automated webhook alerts trigger incident workflows on Mortakaz.

---

## 1. Event-Driven Incident Lifecycle

```
[ Ghaymah Cloud Pod ] ──(OOMKilled Event)──> [ Ghaymah Alertmanager ]
                                                     │
                                                     ▼ Webhook
                                          [ Mortakaz Webhook Listener ]
                                                     │
                                                     ▼
                                          [ Auto-Create Incident Ticket ]
                                                     │
                                                     ▼
                                          [ Trigger HPA / Auto-Remediation ]
```

---

## 2. Mortakaz Webhook Event Schema

When Ghaymah detects a critical alert (e.g. memory saturation > 88% or pod crash), it posts the following JSON payload to Mortakaz:

```json
{
  "event_id": "evt_98410294812",
  "event_type": "CONTAINER_OOM_KILLED",
  "severity": "CRITICAL",
  "qabilah_user": "marwanabdelmoneim",
  "email": "marwantamermo@gmail.com",
  "affected_resource": {
    "platform": "ghaymah.systems",
    "namespace": "production",
    "deployment": "ghaymah-api-deployment",
    "pod_id": "ghaymah-api-7f8d9-x4k21",
    "exit_code": 137,
    "memory_limit_bytes": 536870912,
    "memory_used_bytes": 536870912
  },
  "remediation_action_triggered": "AUTOMATIC_HPA_SCALE_UP",
  "timestamp": "2026-07-27T14:19:00Z"
}
```

---

## 3. Automated Remediation Workflow

1. **Mortakaz Trigger**: Receives `CONTAINER_OOM_KILLED` event.
2. **Auto Scale Execution**: Invokes Ghaymah API endpoint `/v1/deployments/ghaymah-api-deployment/scale` to increase replica count from $N$ to $N+8$.
3. **Paging & Notification**: Pages the SRE on-call engineer via Mortakaz Mobile App and Slack `#sre-incidents` channel.
4. **Postmortem Auto-Drafting**: Generates initial incident timeline data for postmortem analysis.