diff --git a/README.md b/README.md new file mode 100644 index 0000000..c0c9974 --- /dev/null +++ b/README.md @@ -0,0 +1,198 @@ +# ๐ŸŒฉ๏ธ Ghaymah SRE Exam โ€” Amir Kasseb + +> **Site Reliability Engineering** technical exam submission on the [Ghaymah Cloud](https://ghaymah.systems) platform. + +**Candidate:** Amir Mohamed Kasseb + +**Profile:** [qabilah.com/profile/amir-m-kasseb](https://qabilah.com/profile/amir-m-kasseb/professional-profile) + +--- + +## ๐Ÿš€ Live Deployments + +| Service | URL | Description | +|---|---|---| +| ๐Ÿ–ฅ๏ธ **API App** | [gheyma-app-2b88268529f9.hosted.ghaymah.systems](https://gheyma-app-2b88268529f9.hosted.ghaymah.systems) | Go REST API โ€” `/health`, `/metrics` endpoints | +| ๐Ÿ“Š **SRE Dashboard** | [gheyma-dashboard-f50eaa0b0517.hosted.ghaymah.systems](https://gheyma-dashboard-f50eaa0b0517.hosted.ghaymah.systems) | Live monitoring dashboard for the API | +| ๐Ÿ”ญ **Mithal Monitor** | [ghaymah-mithal-monitor-9e02b47e8af8.hosted.ghaymah.systems](https://ghaymah-mithal-monitor-9e02b47e8af8.hosted.ghaymah.systems) | Uptime & latency monitor for mithal.space | + +--- + +## ๐Ÿ“ Repository Structure + +``` +ghaymah-exam-amirkasseb-sre/ +โ”‚ +โ”œโ”€โ”€ q1-deploy-monitor/ # Q1: Container deployment & SRE monitoring +โ”‚ โ”œโ”€โ”€ app/ # Go REST API (main.go, Dockerfile) +โ”‚ โ”œโ”€โ”€ dashboard/ # SRE monitoring dashboard (HTML/CSS/JS + Nginx) +โ”‚ โ”œโ”€โ”€ health-check.sh # Bash health monitoring daemon (30s interval) +โ”‚ โ”œโ”€โ”€ monitor.log # Health check output log +โ”‚ โ””โ”€โ”€ README.md +โ”‚ +โ”œโ”€โ”€ q2-postmortem/ # Q2: Incident postmortem report +โ”‚ โ””โ”€โ”€ postmortem-report.md # OOMKilled outage analysis & recommendations +โ”‚ +โ”œโ”€โ”€ q3-cicd/ # Q3: CI/CD Pipeline +โ”‚ โ”œโ”€โ”€ workflow.yml # GitHub Actions workflow definition +โ”‚ โ””โ”€โ”€ readme.md # Pipeline design explanation (Arabic) +โ”‚ +โ”œโ”€โ”€ q4-scalability/ # Q4: Scalability & load balancing +โ”‚ โ”œโ”€โ”€ calculations.md # Container count calculations & cold-start strategy +โ”‚ โ””โ”€โ”€ architecture.png # Scalability architecture diagram +โ”‚ +โ”œโ”€โ”€ q5-mithal-monitor/ # Q5: mithal.space full-stack monitor +โ”‚ โ”œโ”€โ”€ monitor.py # Python monitoring script (uptime, latency, SSL, DNS) +โ”‚ โ”œโ”€โ”€ index.html # Interactive monitoring dashboard +โ”‚ โ”œโ”€โ”€ data.jsonl / data.csv # Collected monitoring data +โ”‚ โ”œโ”€โ”€ Dockerfile # Container definition +โ”‚ โ””โ”€โ”€ README.md +โ”‚ +โ”œโ”€โ”€ common-mortakaz/ # Mortakaz integration proposals +โ”‚ โ”œโ”€โ”€ integration-1.md # Qumra ร— Ghaymah integration proposal +โ”‚ โ””โ”€โ”€ integration-2.md # Second integration proposal +โ”‚ +โ””โ”€โ”€ common-qabilah/ # Candidate profile + โ””โ”€โ”€ qabilah-profile.txt # Link to professional profile +``` + +--- + +## ๐Ÿ“‹ Questions Summary + +### Q1 โ€” Container Deployment & SRE Monitoring + +Deployed a **Go REST API** to Ghaymah Cloud with: + +- **Multi-stage Dockerfile** (Alpine Linux, non-root user, port 8080) +- **3 endpoints**: `GET /` ยท `GET /health` ยท `GET /metrics` +- **Bash monitoring daemon** (`health-check.sh`) โ€” probes every 30 seconds, logs HTTP status & response time +- **Live SRE dashboard** โ€” real-time latency chart, uptime timeline, request count, auto-refresh every 15s + +```bash +# Verify live endpoints +curl https://gheyma-app-2b88268529f9.hosted.ghaymah.systems/health +curl https://gheyma-app-2b88268529f9.hosted.ghaymah.systems/metrics +``` + +๐Ÿ“‚ [`q1-deploy-monitor/`](./q1-deploy-monitor/) ยท ๐Ÿ“Š [Live Dashboard](https://gheyma-dashboard-f50eaa0b0517.hosted.ghaymah.systems) + +--- + +### Q2 โ€” Incident Postmortem + +Authored a full postmortem for a **45-minute OOMKilled outage**: + +- **Root cause**: Container memory limit set below actual working-set (in-memory cache was never accounted for) +- **Timeline**: Pod crash loop from 10:53 โ†’ resolved at 11:28 after limit increase +- **Recommendations**: Memory-based HPA, 75โ€“80% threshold alerts, restart-count circuit breaker, load testing after stateful changes +- **Ghaymah-specific**: Detection flow using Ghaymah Usage Dashboard & application logs + +๐Ÿ“‚ [`q2-postmortem/postmortem-report.md`](./q2-postmortem/postmortem-report.md) + +--- + +### Q3 โ€” CI/CD Pipeline + +Designed a **GitHub Actions CI/CD pipeline** deploying to Ghaymah Cloud: + +``` +develop โ†’ Build & Test โ†’ Deploy to Staging +main โ†’ Build & Test โ†’ Manual Approval โ†’ Deploy to Production +``` + +- **Build**: Docker image built & health-tested on every push +- **Staging**: Auto-deployed from `develop` branch +- **Production**: Requires manual approval via GitHub Environments before deploy +- **Ghaymah CLI**: Used to authenticate (`gy auth login`) and deploy (`gy resource app launch`) +- **Secrets**: `GHAYMAH_EMAIL` and `GHAYMAH_PW` stored in GitHub Secrets + +๐Ÿ“‚ [`q3-cicd/workflow.yml`](./q3-cicd/workflow.yml) + +--- + +### Q4 โ€” Scalability & Load Balancing + +Calculated infrastructure requirements for **15,000 req/s** with a 30% safety margin: + +``` +Effective load = 15,000 ร— 1.3 = 19,500 req/s +Containers needed = 19,500 รท 500 = 39 containers +โ†’ ~77% capacity utilization per container +``` + +**Cold-start strategies covered:** +| Strategy | Description | +|---|---| +| Predictive Scaling | Spin up containers before threshold is reached | +| Warm Pool | 2โ€“3 standby containers always ready | +| Golden Images | Pre-baked optimized images to cut init time | +| Gradual Traffic Ramp-up | Route load gradually to new containers | +| Readiness Probe | Don't add to load balancer until healthy | + +Also covered **Ghaymah Block Storage** for stateful workloads (persistence, snapshots, low-latency I/O). + +๐Ÿ“‚ [`q4-scalability/calculations.md`](./q4-scalability/calculations.md) + +--- + +### Q5 โ€” Mithal.space Monitor & Dashboard + +Built a **full-stack monitoring application** for `mithal.space`: + +- **Python monitoring script** (`monitor.py`) collecting: Uptime, Latency, SSL certificate expiry, DNS resolution time, Search latency +- **Interactive dashboard** (`index.html`) with real-time charts (Chart.js) +- **Data persistence** in `data.jsonl` and `data.csv` +- **Dockerized**: Single-container deployment with volume mounts for data persistence + +```bash +# Run with Docker Compose +docker compose up -d +# โ†’ Available at http://localhost:8080 + +# Or directly +pip install -r requirements.txt +python monitor.py +``` + +๐Ÿ“‚ [`q5-mithal-monitor/`](./q5-mithal-monitor/) ยท ๐Ÿ”ญ [Live Monitor](https://ghaymah-mithal-monitor-9e02b47e8af8.hosted.ghaymah.systems) + +--- + +### Common โ€” Mortakaz Integration Proposals + +Two integration proposals connecting Arabic SaaS products with Ghaymah Cloud: + +- **Integration 1**: [**Qumra ร— Ghaymah**](./common-mortakaz/integration-1.md) โ€” Store/website builder using Ghaymah as cloud infrastructure layer, monitored via mithal.space +- **Integration 2**: [`integration-2.md`](./common-mortakaz/integration-2.md) โ€” Second integration proposal + +--- + +## ๐Ÿ› ๏ธ Tech Stack + +| Layer | Technology | +|---|---| +| **API** | Go (Golang), Alpine Linux | +| **Containerization** | Docker (multi-stage builds) | +| **CI/CD** | GitHub Actions + Ghaymah CLI | +| **Monitoring Script** | Bash, Python (requests, ssl, dns) | +| **Dashboard** | HTML5, Vanilla CSS, Vanilla JavaScript, Chart.js | +| **Cloud Platform** | [Ghaymah Cloud](https://ghaymah.systems) | + +--- + +## โœ… Verification + +```bash +# API health check +curl -i https://gheyma-app-2b88268529f9.hosted.ghaymah.systems/health +# Expected: HTTP/1.1 200 OK {"status":"healthy"} + +# API metrics +curl -i https://gheyma-app-2b88268529f9.hosted.ghaymah.systems/metrics +# Expected: HTTP/1.1 200 OK {"requests":} +``` + +--- + +> Made with โ˜๏ธ on [Ghaymah Cloud](https://ghaymah.systems)