ghaymah-exam-marwan-abdelmoneim-SRE/common-mortakaz/integration-2.md

Mortakaz Integration Spec 2 — Automated Incident Management & Observability Pipeline

Executive Overview

This specification details the event-driven incident integration between Ghaymah Cloud Monitoring and Mortakaz Incident Response Platform. When container outages (such as OOMKilled events) or threshold breaches occur on Ghaymah, automated webhook alerts trigger incident workflows on Mortakaz.

---

1. Event-Driven Incident Lifecycle

[ Ghaymah Cloud Pod ] ──(OOMKilled Event)──> [ Ghaymah Alertmanager ]
                                                     │
                                                     ▼ Webhook
                                          [ Mortakaz Webhook Listener ]
                                                     │
                                                     ▼
                                          [ Auto-Create Incident Ticket ]
                                                     │
                                                     ▼
                                          [ Trigger HPA / Auto-Remediation ]

---

2. Mortakaz Webhook Event Schema

When Ghaymah detects a critical alert (e.g. memory saturation > 88% or pod crash), it posts the following JSON payload to Mortakaz:

{
  "event_id": "evt_98410294812",
  "event_type": "CONTAINER_OOM_KILLED",
  "severity": "CRITICAL",
  "qabilah_user": "marwanabdelmoneim",
  "email": "marwantamermo@gmail.com",
  "affected_resource": {
    "platform": "ghaymah.systems",
    "namespace": "production",
    "deployment": "ghaymah-api-deployment",
    "pod_id": "ghaymah-api-7f8d9-x4k21",
    "exit_code": 137,
    "memory_limit_bytes": 536870912,
    "memory_used_bytes": 536870912
  },
  "remediation_action_triggered": "AUTOMATIC_HPA_SCALE_UP",
  "timestamp": "2026-07-27T14:19:00Z"
}

---

3. Automated Remediation Workflow

1. Mortakaz Trigger: Receives CONTAINER_OOM_KILLED event.
2. Auto Scale Execution: Invokes Ghaymah API endpoint /v1/deployments/ghaymah-api-deployment/scale to increase replica count from N to N+8.
3. Paging & Notification: Pages the SRE on-call engineer via Mortakaz Mobile App and Slack #sre-incidents channel.
4. Postmortem Auto-Drafting: Generates initial incident timeline data for postmortem analysis.