# 🌩️ Ghaymah SRE Exam — Amir Kasseb

> **Site Reliability Engineering** technical exam submission on the [Ghaymah Cloud](https://ghaymah.systems) platform.

**Candidate:** Amir Mohamed Kasseb

**Profile:** [qabilah.com/profile/amir-m-kasseb](https://qabilah.com/profile/amir-m-kasseb/professional-profile)

---

## 🚀 Live Deployments

| Service | URL | Description |
|---|---|---|
| 🖥️ **API App** | [gheyma-app-2b88268529f9.hosted.ghaymah.systems](https://gheyma-app-2b88268529f9.hosted.ghaymah.systems) | Go REST API — `/health`, `/metrics` endpoints |
| 📊 **SRE Dashboard** | [gheyma-dashboard-f50eaa0b0517.hosted.ghaymah.systems](https://gheyma-dashboard-f50eaa0b0517.hosted.ghaymah.systems) | Live monitoring dashboard for the API |
| 🔭 **Mithal Monitor** | [ghaymah-mithal-monitor-9e02b47e8af8.hosted.ghaymah.systems](https://ghaymah-mithal-monitor-9e02b47e8af8.hosted.ghaymah.systems) | Uptime & latency monitor for mithal.space |

---

## 📁 Repository Structure

```
ghaymah-exam-amirkasseb-sre/
│
├── q1-deploy-monitor/        # Q1: Container deployment & SRE monitoring
│   ├── app/                  # Go REST API (main.go, Dockerfile)
│   ├── dashboard/            # SRE monitoring dashboard (HTML/CSS/JS + Nginx)
│   ├── health-check.sh       # Bash health monitoring daemon (30s interval)
│   ├── monitor.log           # Health check output log
│   └── README.md
│
├── q2-postmortem/            # Q2: Incident postmortem report
│   └── postmortem-report.md  # OOMKilled outage analysis & recommendations
│
├── q3-cicd/                  # Q3: CI/CD Pipeline
│   ├── workflow.yml           # GitHub Actions workflow definition
│   └── readme.md             # Pipeline design explanation (Arabic)
│
├── q4-scalability/           # Q4: Scalability & load balancing
│   ├── calculations.md       # Container count calculations & cold-start strategy
│   └── architecture.png      # Scalability architecture diagram
│
├── q5-mithal-monitor/        # Q5: mithal.space full-stack monitor
│   ├── monitor.py            # Python monitoring script (uptime, latency, SSL, DNS)
│   ├── index.html            # Interactive monitoring dashboard
│   ├── data.jsonl / data.csv # Collected monitoring data
│   ├── Dockerfile            # Container definition
│   └── README.md
│
├── common-mortakaz/          # Mortakaz integration proposals
│   ├── integration-1.md      # Qumra × Ghaymah integration proposal
│   └── integration-2.md      # Second integration proposal
│
└── common-qabilah/           # Candidate profile
    └── qabilah-profile.txt   # Link to professional profile
```

---

## 📋 Questions Summary

### Q1 — Container Deployment & SRE Monitoring

Deployed a **Go REST API** to Ghaymah Cloud with:

- **Multi-stage Dockerfile** (Alpine Linux, non-root user, port 8080)
- **3 endpoints**: `GET /` · `GET /health` · `GET /metrics`
- **Bash monitoring daemon** (`health-check.sh`) — probes every 30 seconds, logs HTTP status & response time
- **Live SRE dashboard** — real-time latency chart, uptime timeline, request count, auto-refresh every 15s

```bash
# Verify live endpoints
curl https://gheyma-app-2b88268529f9.hosted.ghaymah.systems/health
curl https://gheyma-app-2b88268529f9.hosted.ghaymah.systems/metrics
```

📂 [`q1-deploy-monitor/`](./q1-deploy-monitor/) · 📊 [Live Dashboard](https://gheyma-dashboard-f50eaa0b0517.hosted.ghaymah.systems)

---

### Q2 — Incident Postmortem

Authored a full postmortem for a **45-minute OOMKilled outage**:

- **Root cause**: Container memory limit set below actual working-set (in-memory cache was never accounted for)
- **Timeline**: Pod crash loop from 10:53 → resolved at 11:28 after limit increase
- **Recommendations**: Memory-based HPA, 75–80% threshold alerts, restart-count circuit breaker, load testing after stateful changes
- **Ghaymah-specific**: Detection flow using Ghaymah Usage Dashboard & application logs

📂 [`q2-postmortem/postmortem-report.md`](./q2-postmortem/postmortem-report.md)

---

### Q3 — CI/CD Pipeline

Designed a **GitHub Actions CI/CD pipeline** deploying to Ghaymah Cloud:

```
develop → Build & Test → Deploy to Staging
main    → Build & Test → Manual Approval → Deploy to Production
```

- **Build**: Docker image built & health-tested on every push
- **Staging**: Auto-deployed from `develop` branch
- **Production**: Requires manual approval via GitHub Environments before deploy
- **Ghaymah CLI**: Used to authenticate (`gy auth login`) and deploy (`gy resource app launch`)
- **Secrets**: `GHAYMAH_EMAIL` and `GHAYMAH_PW` stored in GitHub Secrets

📂 [`q3-cicd/workflow.yml`](./q3-cicd/workflow.yml)

---

### Q4 — Scalability & Load Balancing

Calculated infrastructure requirements for **15,000 req/s** with a 30% safety margin:

```
Effective load = 15,000 × 1.3 = 19,500 req/s
Containers needed = 19,500 ÷ 500 = 39 containers
→ ~77% capacity utilization per container
```

**Cold-start strategies covered:**
| Strategy | Description |
|---|---|
| Predictive Scaling | Spin up containers before threshold is reached |
| Warm Pool | 2–3 standby containers always ready |
| Golden Images | Pre-baked optimized images to cut init time |
| Gradual Traffic Ramp-up | Route load gradually to new containers |
| Readiness Probe | Don't add to load balancer until healthy |

Also covered **Ghaymah Block Storage** for stateful workloads (persistence, snapshots, low-latency I/O).

📂 [`q4-scalability/calculations.md`](./q4-scalability/calculations.md)

---

### Q5 — Mithal.space Monitor & Dashboard

Built a **full-stack monitoring application** for `mithal.space`:

- **Python monitoring script** (`monitor.py`) collecting: Uptime, Latency, SSL certificate expiry, DNS resolution time, Search latency
- **Interactive dashboard** (`index.html`) with real-time charts (Chart.js)
- **Data persistence** in `data.jsonl` and `data.csv`
- **Dockerized**: Single-container deployment with volume mounts for data persistence

```bash
# Run with Docker Compose
docker compose up -d
# → Available at http://localhost:8080

# Or directly
pip install -r requirements.txt
python monitor.py
```

📂 [`q5-mithal-monitor/`](./q5-mithal-monitor/) · 🔭 [Live Monitor](https://ghaymah-mithal-monitor-9e02b47e8af8.hosted.ghaymah.systems)

---

### Common — Mortakaz Integration Proposals

Two integration proposals connecting Arabic SaaS products with Ghaymah Cloud:

- **Integration 1**: [**Qumra × Ghaymah**](./common-mortakaz/integration-1.md) — Store/website builder using Ghaymah as cloud infrastructure layer, monitored via mithal.space
- **Integration 2**: [`integration-2.md`](./common-mortakaz/integration-2.md) — Second integration proposal

---

## 🛠️ Tech Stack

| Layer | Technology |
|---|---|
| **API** | Go (Golang), Alpine Linux |
| **Containerization** | Docker (multi-stage builds) |
| **CI/CD** | GitHub Actions + Ghaymah CLI |
| **Monitoring Script** | Bash, Python (requests, ssl, dns) |
| **Dashboard** | HTML5, Vanilla CSS, Vanilla JavaScript, Chart.js |
| **Cloud Platform** | [Ghaymah Cloud](https://ghaymah.systems) |

---

## ✅ Verification

```bash
# API health check
curl -i https://gheyma-app-2b88268529f9.hosted.ghaymah.systems/health
# Expected: HTTP/1.1 200 OK  {"status":"healthy"}

# API metrics
curl -i https://gheyma-app-2b88268529f9.hosted.ghaymah.systems/metrics
# Expected: HTTP/1.1 200 OK  {"requests":<count>}
```

---

> Made with ☁️ on [Ghaymah Cloud](https://ghaymah.systems)