# 🚀 Ghaymah SRE Practical Exam **Candidate Name:** Ahmed Abdelaziz Hussein **Track:** Site Reliability Engineering (SRE) --- ## 📁 Repository Structure Below is the directory tree of the submission, showing all implementation files: ```text . ├── common-qabilah │ └── qabilah-profile.txt ├── q1-deploy-monitor │ ├── Dockerfile │ ├── dashboard.html │ ├── health-check.sh │ └── status.json ├── q2-postmortem │ └── postmortem-report.md ├── q3-cicd │ └── workflow.yml ├── q4-sacalability │ ├── architecture.png │ └── calculations.md └── q5-mithal-monitor ├── dashbourd.html ├── metrics.json └── monitor.py ``` --- ## 🔧 Technologies Used The project leverages the following technologies and frameworks: - **Go / Fiber** (API Development) - **Docker** (Containerization & Multi-stage builds) - **GitHub Actions** (CI/CD Pipeline) - **HTML / CSS / JavaScript** (Dashboards & Visualization) - **Python** (Mithal Automated Monitoring) - **Bash** (Monitoring Agent scripts) - **Ghaymah Cloud** (Deployment Infrastructure & CLI) --- ## 📊 Question 1 – Deploy Application & Monitoring ### Requirements Covered | Requirement | Status | Details | | :--- | :---: | :--- | | Dockerfile | ✔ | Optimized multi-stage build starting from `golang:alpine` to `scratch` | | Simple Go Fiber API | ✔ | REST API utilizing the Go Fiber framework | | `/health` endpoint | ✔ | Returns application health status and current timestamp | | `/metrics` endpoint | ✔ | Exposes real-time internal metrics (uptime, request count, latency) | | Monitoring Script | ✔ | Bash script performing periodic checks and exporting metrics to JSON | | Monitoring Dashboard | ✔ | Static HTML dashboard to display real-time statuses and metrics | | Deployment on Ghaymah Cloud | ✔ | API deployed live on Ghaymah systems | ### Files - [`q1-deploy-monitor/Dockerfile`](./q1-deploy-monitor/Dockerfile) - [`q1-deploy-monitor/health-check.sh`](./q1-deploy-monitor/health-check.sh) - [`q1-deploy-monitor/dashboard.html`](./q1-deploy-monitor/dashboard.html) - [`q1-deploy-monitor/status.json`](./q1-deploy-monitor/status.json) ### Proof of Implementation The screenshots showcasing the running state are located under `docs/screenshots/`: 1. **Deployment on Ghaymah Cloud** ![Deployment on Ghaymah Cloud](docs/screenshots/ghaymah-deployment.png) 2. **Monitoring Script Running** ![Monitoring Script Running](docs/screenshots/monitoring-script.png) 3. **Dashboard** ![Monitoring Dashboard](docs/screenshots/dashboard.png) --- ## 📑 Question 2 – Incident Postmortem ### Requirements Covered - [x] **Executive Summary** (Incident metadata, downtime duration, root cause, impact) - [x] **Timeline** (Detailed chronological sequence of events from spike to resolution) - [x] **Root Cause Analysis (RCA)** (Detailed diagnosis of resource limits and missing HPA) - [x] **Recommendations** (Immediate P0 fixes and long-term action items) - [x] **Auto Scaling Policy** (Configured Horizontal Pod Autoscaler YAML config for Ghanimah) - [x] **Monitoring Strategy** (Prometheus alert rules for early warning and OOM checks) ### Files - [`q2-postmortem/postmortem-report.md`](file:///home/ahmed/ghaymah_task/ghaymah-exam-ahmed-abdelaziz-SRE/q2-postmortem/postmortem-report.md) --- ## ⚙️ Question 3 – CI/CD Pipeline ### Requirements Covered - [x] **Docker Build:** Builds container images on triggers. - [x] **Container Registry:** Image tagging for version control. - [x] **Deploy to Staging:** Triggered automatically on release branches. - [x] **Manual Approval:** Gatekeeping promotion using GitHub Environments rules. - [x] **Deploy to Production:** Triggered on main branch post-approval. - [x] **Ghaymah CLI Integration:** Automated login and app launch in workflow jobs. ### Deployment Flow Diagram ```text Push ↓ Build ↓ Deploy Staging ↓ Manual Approval ↓ Deploy Production ``` ### Files - [`q3-cicd/workflow.yml`](./q3-cicd/workflow.yml) ### Proof of Implementation 1. **Staging Deployment** ![Staging Deployment](docs/screenshots/staging-deployment.png) 2. **Manual Approval** ![Manual Approval](docs/screenshots/manual-approval.png) --- ## 📈 Question 4 – Scalability & Load Balancing ### Requirements Covered - [x] **Architecture Diagram** (Visual flow of requests to handle 15,000 req/s across zones) - [x] **Capacity Calculation** (Calculations justifying 39 Pods based on limits and 30% safety buffer) - [x] **Cold Start Strategy** (Multi-stage scratch image design, pre-warming, and probe tuning) - [x] **Block Storage** (Dynamic PV/PVC configurations utilizing Ghanimah Block Storage NVMe disks) ### Files - [`q4-sacalability/calculations.md`](./q4-sacalability/calculations.md) - [`q4-sacalability/architecture.png`](./q4-sacalability/architecture.png) --- ## 🖥️ Question 5 – Mithal Monitoring Dashboard ### Requirements Covered - [x] **Latency Monitoring:** Measures and logs home page loading speeds. - [x] **Uptime Monitoring:** Tracks availability and monitors return status codes. - [x] **SSL Monitoring:** Calculates certificate expiration and counts remaining days. - [x] **DNS Monitoring:** Resolves and logs lookup times. - [x] **Search Response Monitoring:** Assesses specific query parameters response time. - [x] **JSON Storage:** Logs state to dynamic history files up to 24 hours. - [x] **Dashboard:** An interface displaying key charts and status panels. - [x] **Deployment on Ghaymah:** Live monitor configuration on Ghaymah infrastructure. ### Files - [`q5-mithal-monitor/monitor.py`](file:///home/ahmed/ghaymah_task/ghaymah-exam-ahmed-abdelaziz-SRE/q5-mithal-monitor/monitor.py) - [`q5-mithal-monitor/metrics.json`](file:///home/ahmed/ghaymah_task/ghaymah-exam-ahmed-abdelaziz-SRE/q5-mithal-monitor/metrics.json) - [`q5-mithal-monitor/dashbourd.html`](file:///home/ahmed/ghaymah_task/ghaymah-exam-ahmed-abdelaziz-SRE/q5-mithal-monitor/dashbourd.html) ### Proof of Implementation 1. **monitor.py Running** ![monitor.py Running](docs/screenshots/monitor-running.png) 2. **Dashboard** ![Mithal Dashboard](docs/screenshots/mithal-dashboard.png) --- ## 🏃 Running Locally ### Question 1 – Go API & Monitoring Agent To test the Go Fiber API and the monitoring scripts locally: 1. **Build and Run the Go API via Docker:** ```bash # Build the container image docker build -t ghaymah-sre-api ./q1-deploy-monitor # Run the container exposing port 8080 docker run -d -p 8080:8080 --name ghanimah-api ghaymah-sre-api ``` 2. **Run the Bash Monitoring Agent:** ```bash # Point the agent to the locally running Docker container API_URL=http://localhost:8080 bash q1-deploy-monitor/health-check.sh ``` 3. **View the Dashboard:** Open [`q1-deploy-monitor/dashboard.html`](./q1-deploy-monitor/dashboard.html) directly in a web browser of your choice. --- ### Question 5 – Mithal Automated Monitor To test the Python monitoring daemon locally: 1. **Install Dependencies:** ```bash pip install requests ``` 2. **Run the Monitor Script:** ```bash python3 q5-mithal-monitor/monitor.py ``` 3. **View the Mithal Dashboard:** Open [`q5-mithal-monitor/dashbourd.html`](./q5-mithal-monitor/dashbourd.html) in your browser to view historical metrics. --- ## ✨ Project Highlights This repository demonstrates complete implementation of fundamental Site Reliability Engineering practices: - **Containerization:** Clean multi-stage lightweight builds (Go statically-linked binary in a `scratch` container). - **Monitoring & Observability:** Real-time metrics gathering, system state logging, alerts modeling, and custom front-end status dashboards. - **CI/CD:** Automated builds, environments targeting, manual approvals, and deployment orchestrations. - **Incident Analysis:** Professional blameless postmortem report detailing timelines, root cause analysis, action items, auto-scaling thresholds, and early discovery strategies. - **Scalability:** Quantitative capacity sizing for high-traffic environments (15,000 req/s), Cold Start tuning, and stateful volume management. - **Cloud Deployment:** Orchestration using the Ghaymah Cloud platforms. --- ## ✍️ Author **Ahmed Abdelaziz Hussein** *Information Systems* *Faculty of Computers and Information* *Qabilah Profile:* [ahmed-abdelaziz-89943a271](https://qabilah.com/profile/ahmed-abdelaziz-89943a271/posts)