# ☁️ Ghaymah Cloud SRE & SecOps Exam Repository

**Repository Name**: `ghaymah-exam-marwan-abdelmoneim-SRE`  
**Candidate Name**: Marwan Abdelmoneim  
**Qabilah Username**: `marwanabdelmoneim` *(Qabilah User Account)*  
**Email Address**: `marwantamermo@gmail.com`  
**Track**: SRE — Site Reliability Engineering  
**Target Platform**: Ghaymah Cloud (`ghaymah.systems`) / Qabilah Platform  
**Total Grade**: 100 / 100 Marks  

---

## 📂 Repository Directory Architecture

```
ghaymah-exam-marwan-abdelmoneim-SRE/
├── q1-deploy-monitor/
│   ├── Dockerfile             # Multi-stage production Python 3.11 Dockerfile with non-root security context
│   ├── health-check.sh        # Bash monitoring script checking /healthz every 30 seconds
│   └── dashboard.html         # Real-time HTML/CSS/JS Glassmorphism container monitor
├── q2-postmortem/
│   └── postmortem-report.md   # SEV-1 45-min OOM outage report, 5-Whys, HPA spec & early detection runbook
├── q3-cicd/
│   └── workflow.yml           # GitHub Actions workflow with Trivy security scanning, manual approval & Ghaymah CLI
├── q4-scalability/
│   ├── architecture.png       # 15,000 req/s System Architecture Diagram
│   └── calculations.md        # Capacity math (39 container derivation), cold starts & NVMe Block Storage
├── q5-mithal-monitor/
│   ├── monitor.py             # Python telemetry engine collecting Latency, Uptime, SSL, DNS & Search RTT
│   └── dashboard.html         # Interactive mithal.space monitoring dashboard
├── common-mortakaz/
│   ├── integration-1.md       # Mortakaz Telemetry & Security Hub Integration Spec
│   └── integration-2.md       # Mortakaz Automated Incident & Webhook Integration Spec
├── common-qabilah/
│   └── qabilah-profile.txt    # Official Qabilah Platform Candidate Credentials & Profile
└── README.md                  # Project Master Documentation & Submission Index
```

---

## 📋 Comprehensive Module Breakdown & Solution Guide

### 1️⃣ Question 1: Deploy & Monitor App on Ghaymah (`q1-deploy-monitor/`) [20 Marks]
- **Dockerfile**: Multi-stage `python:3.11-slim` build with non-root user `appuser` (UID 10001), healthcheck instructions (`HEALTHCHECK`), and minimal attack surface.
- **health-check.sh**: Executable bash script querying `http://localhost:8000/healthz` every 30 seconds, measuring round-trip latency, appending to log store, and triggering PagerDuty alerts on failure.
- **dashboard.html**: Self-contained glassmorphism Web UI featuring real-time health indicator, response time trend line chart, total request counter, and recent logs table.

### 2️⃣ Question 2: Incident Analysis — Postmortem & Auto-Scaling (`q2-postmortem/`) [20 Marks]
- **postmortem-report.md**:
  - Full incident timeline of the 45-minute OOMKilled outage.
  - Root Cause Analysis (RCA) & 5-Whys technique identifying memory leak & improper cgroup bounds.
  - **Ghaymah HPA Manifest**: Dual-metric scaling rule triggering at 70% memory and 75% CPU with zero scale-up delay.
  - **Early Detection Runbook**: Prometheus alerts monitoring `container_memory_working_set_bytes` > 75% and positive memory growth derivatives (`deriv > 0`).

### 3️⃣ Question 3: CI/CD Pipeline on Ghaymah (`q3-cicd/`) [20 Marks]
- **workflow.yml**:
  - GitHub Actions pipeline performing unit testing, Trivy container vulnerability scanning, image push to `registry.ghaymah.systems`.
  - Automated deployment to `staging.ghaymah.systems`.
  - **Manual Approval Gate**: Enforces senior engineer review before executing canary deployment (10% -> 100%) to `api.ghaymah.systems` via Ghaymah CLI.
  - Detailed architectural breakdown comparing Staging vs. Production and Ghaymah CLI commands.

### 4️⃣ Question 4: Scalability & Load Balancing (`q4-scalability/`) [20 Marks]
- **calculations.md**:
  - **Capacity Derivation**: Peak load = $15,000 \times 1.30 = 19,500 \text{ req/s}$. Total required containers = $\frac{19,500}{500} = \mathbf{39 \text{ active containers}}$ (13 per AZ across 3 Availability Zones).
  - **Cold Start Mitigation**: Distroless images (< 45MB), node image pre-caching, +10% warm pod pool, pre-warmed connection pools.
  - **Stateful Workloads**: Ghaymah NVMe Block Storage (`ghaymah-block-nvme`), ReadWriteOnce (RWO) PVCs for PostgreSQL WAL streaming.
- **architecture.png**: Generated high-resolution architecture diagram.

### 5️⃣ Question 5: Monitoring Dashboard for `mithal.space` (`q5-mithal-monitor/`) [20 Marks]
- **monitor.py**: Python telemetry script capturing HTTP Latency, Uptime (200 OK), SSL Certificate validity & days remaining, DNS lookup duration, and Search query RTT against `https://mithal.space`. Output saved to `metrics.json` and `metrics.csv`.
- **dashboard.html**: Interactive dashboard showing 24h Uptime gauge, Latency trend chart, SSL countdown badge, and historical checks table.

### 6️⃣ Platform Integrations (`common-mortakaz/` & `common-qabilah/`)
- **integration-1.md**: Centralized SRE Telemetry & Security Hub integration schema with Mortakaz Platform via mTLS.
- **integration-2.md**: Event-driven Webhook incident response automation connecting Ghaymah Alertmanager with Mortakaz.
- **qabilah-profile.txt**: Candidate profile linking `marwanabdelmoneim` account with `marwantamermo@gmail.com`.

---

## 🚀 Execution & Quick Start Instructions

### Run Task 1 Health Monitoring Script
```bash
cd q1-deploy-monitor
chmod +x health-check.sh
./health-check.sh
```

### Run Task 5 Telemetry Collector for `mithal.space`
```bash
cd q5-mithal-monitor
python monitor.py
```

---
*Submitted by: Marwan Abdelmoneim (`marwanabdelmoneim`)*  
*Ghaymah Cloud Platform & Qabilah Platform Exam*