59 أسطر
2.3 KiB
Markdown
59 أسطر
2.3 KiB
Markdown
# Mortakaz Integration Spec 2 — Automated Incident Management & Observability Pipeline
|
|
|
|
## Executive Overview
|
|
|
|
This specification details the event-driven incident integration between **Ghaymah Cloud Monitoring** and **Mortakaz Incident Response Platform**. When container outages (such as `OOMKilled` events) or threshold breaches occur on Ghaymah, automated webhook alerts trigger incident workflows on Mortakaz.
|
|
|
|
---
|
|
|
|
## 1. Event-Driven Incident Lifecycle
|
|
|
|
```
|
|
[ Ghaymah Cloud Pod ] ──(OOMKilled Event)──> [ Ghaymah Alertmanager ]
|
|
│
|
|
▼ Webhook
|
|
[ Mortakaz Webhook Listener ]
|
|
│
|
|
▼
|
|
[ Auto-Create Incident Ticket ]
|
|
│
|
|
▼
|
|
[ Trigger HPA / Auto-Remediation ]
|
|
```
|
|
|
|
---
|
|
|
|
## 2. Mortakaz Webhook Event Schema
|
|
|
|
When Ghaymah detects a critical alert (e.g. memory saturation > 88% or pod crash), it posts the following JSON payload to Mortakaz:
|
|
|
|
```json
|
|
{
|
|
"event_id": "evt_98410294812",
|
|
"event_type": "CONTAINER_OOM_KILLED",
|
|
"severity": "CRITICAL",
|
|
"qabilah_user": "marwanabdelmoneim",
|
|
"email": "marwantamermo@gmail.com",
|
|
"affected_resource": {
|
|
"platform": "ghaymah.systems",
|
|
"namespace": "production",
|
|
"deployment": "ghaymah-api-deployment",
|
|
"pod_id": "ghaymah-api-7f8d9-x4k21",
|
|
"exit_code": 137,
|
|
"memory_limit_bytes": 536870912,
|
|
"memory_used_bytes": 536870912
|
|
},
|
|
"remediation_action_triggered": "AUTOMATIC_HPA_SCALE_UP",
|
|
"timestamp": "2026-07-27T14:19:00Z"
|
|
}
|
|
```
|
|
|
|
---
|
|
|
|
## 3. Automated Remediation Workflow
|
|
|
|
1. **Mortakaz Trigger**: Receives `CONTAINER_OOM_KILLED` event.
|
|
2. **Auto Scale Execution**: Invokes Ghaymah API endpoint `/v1/deployments/ghaymah-api-deployment/scale` to increase replica count from $N$ to $N+8$.
|
|
3. **Paging & Notification**: Pages the SRE on-call engineer via Mortakaz Mobile App and Slack `#sre-incidents` channel.
|
|
4. **Postmortem Auto-Drafting**: Generates initial incident timeline data for postmortem analysis.
|