# 🚀 Ghaymah SRE Practical Exam

**Candidate Name:** Ahmed Abdelaziz Hussein  
**Track:** Site Reliability Engineering (SRE)  

---

## 📁 Repository Structure

Below is the directory tree of the submission, showing all implementation files:

```text
.
├── common-qabilah
│   └── qabilah-profile.txt
├── q1-deploy-monitor
│   ├── Dockerfile
│   ├── dashboard.html
│   ├── health-check.sh
│   └── status.json
├── q2-postmortem
│   └── postmortem-report.md
├── q3-cicd
│   └── workflow.yml
├── q4-sacalability
│   ├── architecture.png
│   └── calculations.md
└── q5-mithal-monitor
    ├── dashbourd.html
    ├── metrics.json
    └── monitor.py
```

---

## 🔧 Technologies Used

The project leverages the following technologies and frameworks:
- **Go / Fiber** (API Development)
- **Docker** (Containerization & Multi-stage builds)
- **GitHub Actions** (CI/CD Pipeline)
- **HTML / CSS / JavaScript** (Dashboards & Visualization)
- **Python** (Mithal Automated Monitoring)
- **Bash** (Monitoring Agent scripts)
- **Ghaymah Cloud** (Deployment Infrastructure & CLI)

---

## 📊 Question 1 – Deploy Application & Monitoring

### Requirements Covered
| Requirement | Status | Details |
| :--- | :---: | :--- |
| Dockerfile | ✔ | Optimized multi-stage build starting from `golang:alpine` to `scratch` |
| Simple Go Fiber API | ✔ | REST API utilizing the Go Fiber framework |
| `/health` endpoint | ✔ | Returns application health status and current timestamp |
| `/metrics` endpoint | ✔ | Exposes real-time internal metrics (uptime, request count, latency) |
| Monitoring Script | ✔ | Bash script performing periodic checks and exporting metrics to JSON |
| Monitoring Dashboard | ✔ | Static HTML dashboard to display real-time statuses and metrics |
| Deployment on Ghaymah Cloud | ✔ | API deployed live on Ghaymah systems |

### Files
- [`q1-deploy-monitor/Dockerfile`](./q1-deploy-monitor/Dockerfile)
- [`q1-deploy-monitor/health-check.sh`](./q1-deploy-monitor/health-check.sh)
- [`q1-deploy-monitor/dashboard.html`](./q1-deploy-monitor/dashboard.html)
- [`q1-deploy-monitor/status.json`](./q1-deploy-monitor/status.json)

### Proof of Implementation
The screenshots showcasing the running state are located under `docs/screenshots/`:

1. **Deployment on Ghaymah Cloud**
   ![Deployment on Ghaymah Cloud](docs/screenshots/ghaymah-deployment.png)

2. **Monitoring Script Running**
   ![Monitoring Script Running](docs/screenshots/monitoring-script.png)

3. **Dashboard**
   ![Monitoring Dashboard](docs/screenshots/dashboard.png)
---

## 📑 Question 2 – Incident Postmortem

### Requirements Covered
- [x] **Executive Summary** (Incident metadata, downtime duration, root cause, impact)
- [x] **Timeline** (Detailed chronological sequence of events from spike to resolution)
- [x] **Root Cause Analysis (RCA)** (Detailed diagnosis of resource limits and missing HPA)
- [x] **Recommendations** (Immediate P0 fixes and long-term action items)
- [x] **Auto Scaling Policy** (Configured Horizontal Pod Autoscaler YAML config for Ghanimah)
- [x] **Monitoring Strategy** (Prometheus alert rules for early warning and OOM checks)

### Files
- [`q2-postmortem/postmortem-report.md`](file:///home/ahmed/ghaymah_task/ghaymah-exam-ahmed-abdelaziz-SRE/q2-postmortem/postmortem-report.md)

---

## ⚙️ Question 3 – CI/CD Pipeline

### Requirements Covered
- [x] **Docker Build:** Builds container images on triggers.
- [x] **Container Registry:** Image tagging for version control.
- [x] **Deploy to Staging:** Triggered automatically on release branches.
- [x] **Manual Approval:** Gatekeeping promotion using GitHub Environments rules.
- [x] **Deploy to Production:** Triggered on main branch post-approval.
- [x] **Ghaymah CLI Integration:** Automated login and app launch in workflow jobs.

### Deployment Flow Diagram
```text
Push
  ↓
Build
  ↓
Deploy Staging
  ↓
Manual Approval
  ↓
Deploy Production
```

### Files
- [`q3-cicd/workflow.yml`](./q3-cicd/workflow.yml)

### Proof of Implementation
1. **Staging Deployment**
   ![Staging Deployment](docs/screenshots/staging-deployment.png)

2. **Manual Approval**
   ![Manual Approval](docs/screenshots/manual-approval.png)
---

## 📈 Question 4 – Scalability & Load Balancing

### Requirements Covered
- [x] **Architecture Diagram** (Visual flow of requests to handle 15,000 req/s across zones)
- [x] **Capacity Calculation** (Calculations justifying 39 Pods based on limits and 30% safety buffer)
- [x] **Cold Start Strategy** (Multi-stage scratch image design, pre-warming, and probe tuning)
- [x] **Block Storage** (Dynamic PV/PVC configurations utilizing Ghanimah Block Storage NVMe disks)

### Files
- [`q4-sacalability/calculations.md`](./q4-sacalability/calculations.md)
- [`q4-sacalability/architecture.png`](./q4-sacalability/architecture.png)

---

## 🖥️ Question 5 – Mithal Monitoring Dashboard

### Requirements Covered
- [x] **Latency Monitoring:** Measures and logs home page loading speeds.
- [x] **Uptime Monitoring:** Tracks availability and monitors return status codes.
- [x] **SSL Monitoring:** Calculates certificate expiration and counts remaining days.
- [x] **DNS Monitoring:** Resolves and logs lookup times.
- [x] **Search Response Monitoring:** Assesses specific query parameters response time.
- [x] **JSON Storage:** Logs state to dynamic history files up to 24 hours.
- [x] **Dashboard:** An interface displaying key charts and status panels.
- [x] **Deployment on Ghaymah:** Live monitor configuration on Ghaymah infrastructure.

### Files
- [`q5-mithal-monitor/monitor.py`](file:///home/ahmed/ghaymah_task/ghaymah-exam-ahmed-abdelaziz-SRE/q5-mithal-monitor/monitor.py)
- [`q5-mithal-monitor/metrics.json`](file:///home/ahmed/ghaymah_task/ghaymah-exam-ahmed-abdelaziz-SRE/q5-mithal-monitor/metrics.json)
- [`q5-mithal-monitor/dashbourd.html`](file:///home/ahmed/ghaymah_task/ghaymah-exam-ahmed-abdelaziz-SRE/q5-mithal-monitor/dashbourd.html)

### Proof of Implementation
1. **monitor.py Running**
   ![monitor.py Running](docs/screenshots/monitor-running.png)

2. **Dashboard**
   ![Mithal Dashboard](docs/screenshots/mithal-dashboard.png)
---

## 🏃 Running Locally

### Question 1 – Go API & Monitoring Agent
To test the Go Fiber API and the monitoring scripts locally:

1. **Build and Run the Go API via Docker:**
   ```bash
   # Build the container image
   docker build -t ghaymah-sre-api ./q1-deploy-monitor

   # Run the container exposing port 8080
   docker run -d -p 8080:8080 --name ghanimah-api ghaymah-sre-api
   ```
2. **Run the Bash Monitoring Agent:**
   ```bash
   # Point the agent to the locally running Docker container
   API_URL=http://localhost:8080 bash q1-deploy-monitor/health-check.sh
   ```
3. **View the Dashboard:**
   Open [`q1-deploy-monitor/dashboard.html`](./q1-deploy-monitor/dashboard.html) directly in a web browser of your choice.

---

### Question 5 – Mithal Automated Monitor
To test the Python monitoring daemon locally:

1. **Install Dependencies:**
   ```bash
   pip install requests
   ```
2. **Run the Monitor Script:**
   ```bash
   python3 q5-mithal-monitor/monitor.py
   ```
3. **View the Mithal Dashboard:**
   Open [`q5-mithal-monitor/dashbourd.html`](./q5-mithal-monitor/dashbourd.html) in your browser to view historical metrics.

---

## ✨ Project Highlights

This repository demonstrates complete implementation of fundamental Site Reliability Engineering practices:
- **Containerization:** Clean multi-stage lightweight builds (Go statically-linked binary in a `scratch` container).
- **Monitoring & Observability:** Real-time metrics gathering, system state logging, alerts modeling, and custom front-end status dashboards.
- **CI/CD:** Automated builds, environments targeting, manual approvals, and deployment orchestrations.
- **Incident Analysis:** Professional blameless postmortem report detailing timelines, root cause analysis, action items, auto-scaling thresholds, and early discovery strategies.
- **Scalability:** Quantitative capacity sizing for high-traffic environments (15,000 req/s), Cold Start tuning, and stateful volume management.
- **Cloud Deployment:** Orchestration using the Ghaymah Cloud platforms.

---

## ✍️ Author

**Ahmed Abdelaziz Hussein**  
*Information Systems*  
*Faculty of Computers and Information*  
*Qabilah Profile:* [ahmed-abdelaziz-89943a271](https://qabilah.com/profile/ahmed-abdelaziz-89943a271/posts)
