Edit README

README.md

@@ -0,0 +1,175 @@
+# Ghaymah SRE Exam Submission
+
+This repository contains a completed set of SRE deliverables for the Ghaymah exam. It includes a containerized Flask API with health monitoring, an incident postmortem, a CI/CD workflow, scalability calculations, and an independent monitoring dashboard for `https://mithal.space`.
+
+## Contents
+
+| Area | Location | Summary |
+| --- | --- | --- |
+| Q1 — Deployment & monitoring | [`q1-deploy-monitor/`](q1-deploy-monitor/) | Docker image, health-check script, and live API dashboard |
+| Q2 — Postmortem | [`q2-postmortem/postmortem-report.md`](q2-postmortem/postmortem-report.md) | SEV-1 OOMKilled incident analysis and prevention plan |
+| Q3 — CI/CD | [`q3-cicd/`](q3-cicd/) | GitHub Actions workflow and environment/deployment guidance |
+| Q4 — Scalability | [`q4-scalability/`](q4-scalability/) | Capacity calculation, cold-start strategy, and architecture diagram |
+| Q5 — Mithal monitor | [`q5-mithal-monitor/`](q5-mithal-monitor/) | Website monitor, metric history, and monitoring dashboard |
+| Community profile | [`common-qabilah/qabilah-profile.md`](common-qabilah/qabilah-profile.md) | Qabilah profile link |
+
+## Prerequisites
+
+- Python 3.12+ for the main API and monitor
+- Docker (optional, for containerized runs)
+- `curl` to use the continuous health-check script
+- Network access to `https://mithal.space` when running the Mithal monitor
+
+## Q1 — API deployment and health monitoring
+
+The root Flask application provides a lightweight service health endpoint and a browser dashboard. It tracks its process uptime and the number of requests handled since startup.
+
+**App URL on Ghaymah Systems: [https://ghaymah-sre-api-320ed50fd626.hosted.ghaymah.systems](https://ghaymah-sre-api-320ed50fd626.hosted.ghaymah.systems)**
+
+### Run locally
+
+```bash
+python3 -m venv .venv
+source .venv/bin/activate
+pip install -r requirements.txt
+python app.py
+```
+
+The service listens on `0.0.0.0:5000`.
+
+| URL | Purpose |
+| --- | --- |
+| `http://localhost:5000/` | Service landing page |
+| `http://localhost:5000/health` | JSON health status, uptime, and request count |
+| `http://localhost:5000/dashboard` | Browser dashboard polling the health endpoint every 30 seconds |
+
+Example health response:
+
+```json
+{
+  "requests": 1,
+  "status": "healthy",
+  "uptime": 0.02
+}
+```
+
+### Run with Docker
+
+Build from the repository root so the Dockerfile can copy the shared application files:
+
+```bash
+docker build -f q1-deploy-monitor/Dockerfile -t ghaymah-sre-api .
+docker run --rm -p 5000:5000 ghaymah-sre-api
+```
+
+### Continuous health check
+
+With the API running on port 5000, run the following in a second terminal:
+
+```bash
+bash q1-deploy-monitor/health-check.sh
+```
+
+The script requests `/health` every 30 seconds and prints the UTC-independent local timestamp, availability state, and request latency. Stop it with `Ctrl+C`.
+
+## Q2 — OOMKilled postmortem
+
+The [postmortem report](q2-postmortem/postmortem-report.md) documents a 45-minute SEV-1 outage caused by repeated container OOM kills under increased load. It covers:
+
+- Incident impact and a detailed recovery timeline
+- Root cause and contributing factors
+- Immediate remediation and prevention measures
+- HPA policy recommendations and early-detection metrics/alerts
+
+## Q3 — CI/CD
+
+[`q3-cicd/workflow.yml`](q3-cicd/workflow.yml) defines a GitHub Actions pipeline triggered by pushes to `main`:
+
+```text
+Build image → Authenticate to Ghaymah registry → Push image → Deploy staging → Deploy production
+```
+
+The build uses `q1-deploy-monitor/Dockerfile` and tags the image with the Git commit SHA. The registry steps require these repository secrets:
+
+| Secret | Purpose |
+| --- | --- |
+| `GHAYMAH_USERNAME` | Ghaymah registry account name |
+| `GHAYMAH_TOKEN` | Ghaymah registry authentication token |
+
+Staging is represented by the `staging` GitHub environment. Production uses the `production` environment; configure its environment protection rules to require manual approval before the production job starts.
+
+See the [CI/CD documentation](q3-cicd/README.md) for the staging/production distinction and Ghaymah CLI commands.
+
+## Q4 — Scalability and load balancing
+
+The capacity plan assumes 15,000 requests per second, a nominal capacity of 500 requests per second per container, and a 30% safety margin:
+
+```text
+Effective capacity/container = 500 × 70% = 350 requests/second
+Required containers          = ceil(15,000 ÷ 350) = 43
+```
+
+The plan recommends warm containers, autoscaling at CPU above 70%, pre-pulled images, and readiness/liveness checks before load-balancer registration. It also describes suitable persistent-workload use cases for Ghaymah Block Storage.
+
+![Scalability architecture](q4-scalability/architecture.png)
+
+For the complete rationale, see [the calculations](q4-scalability/calculations.md).
+
+## Q5 — Mithal monitoring dashboard
+
+This is a separate Flask service that monitors `https://mithal.space` every 60 seconds. Each collection records:
+
+**App URL on Ghaymah Systems: [https://mithal-dashboard-e37761be7d74.hosted.ghaymah.systems](https://mithal-dashboard-e37761be7d74.hosted.ghaymah.systems)**
+
+- HTTP status and response latency
+- DNS lookup latency
+- TLS certificate expiration date and days remaining
+- A second endpoint-request latency, shown as “Search” in the dashboard
+- Uptime percentage based on successful (`200`) checks
+
+Metric samples are retained in `metrics.json`: the last 1,440 samples are kept internally, while the API returns the latest 10 history entries and latest 60 latency values for the chart. The data file is generated at runtime and is not included in the repository.
+
+### Run locally
+
+```bash
+cd q5-mithal-monitor
+python3 -m venv .venv
+source .venv/bin/activate
+pip install -r requirements.txt
+python app.py
+```
+
+Open `http://localhost:8080/` for the dashboard, or retrieve raw metrics from `http://localhost:8080/metrics.json`.
+
+### Run with Docker
+
+Build using the monitor directory as the build context:
+
+```bash
+docker build -t mithal-monitor q5-mithal-monitor
+docker run --rm -p 8080:8080 mithal-monitor
+```
+
+> The dashboard loads Chart.js from a public CDN, so it needs browser internet access to render the latency chart.
+
+## Project layout
+
+```text
+.
+├── app.py                         # Main Flask API
+├── requirements.txt               # Main API dependencies
+├── q1-deploy-monitor/             # API Dockerfile, dashboard, health checker
+├── q2-postmortem/                 # Incident postmortem
+├── q3-cicd/                       # GitHub Actions workflow and CI/CD notes
+├── q4-scalability/                # Capacity analysis and architecture image
+├── q5-mithal-monitor/             # Independent external-site monitor
+└── common-qabilah/                # Community profile
+```
+
+## Notes and limitations
+
+- Both Flask applications use the development server, which is appropriate for this exercise. A production deployment should use a production WSGI server such as Gunicorn (already listed in the root requirements) behind a reverse proxy/load balancer.
+- The main API’s request counter and uptime are in-memory and reset whenever the application restarts.
+- The Mithal monitor writes metrics to its local container filesystem. Mount persistent storage if history must survive container replacement.
+- The deployment jobs currently contain placeholder deployment commands; connect them to the target Ghaymah environment before using the workflow for a live release.
+

common-qabilah/qabilah-profile.md

@@ -0,0 +1,4 @@
+## Community
+
+My Qabilah profile:
+https://qabilah.com/profile/mohamedsale7/posts

q4-scalability/calculations.md

@@ -0,0 +1,62 @@
+# Q4 - Scalability & Load Balancing
+
+## 1. Architecture
+
+See `architecture.png`.
+
+---
+
+## 2. Container Calculation
+
+### Given
+
+- Required traffic = **15,000 requests/second**
+- One container capacity = **500 requests/second**
+- Safety margin = **30%**
+
+### Effective Capacity
+
+Effective capacity per container:
+
+500 × 70% = **350 requests/second**
+
+### Required Containers
+
+15000 ÷ 350 = **42.86**
+
+Rounded up:
+
+**43 Containers**
+
+---
+
+## 3. Cold Start Strategy
+
+To reduce startup latency:
+
+- Keep 2–3 warm containers running.
+- Enable Auto Scaling when CPU > 70%.
+- Pre-pull Docker images on worker nodes.
+- Use Readiness and Liveness probes.
+- Register new containers with the Load Balancer only after passing health checks.
+
+---
+
+## 4. Ghaymah Block Storage
+
+Ghaymah Block Storage provides persistent storage for stateful workloads.
+
+Examples:
+
+- PostgreSQL
+- MySQL
+- MongoDB
+- Redis persistence
+- User uploaded files
+
+Benefits:
+
+- Persistent storage after container restart.
+- High availability.
+- Supports snapshots and backups.
+- Suitable for databases and stateful applications.

q5-mithal-monitor/Dockerfile

@@ -0,0 +1,13 @@
+FROM python:3.12-slim
+
+WORKDIR /app
+
+COPY requirements.txt .
+
+RUN pip install --no-cache-dir -r requirements.txt
+
+COPY . .
+
+EXPOSE 8080
+
+CMD ["python3","app.py"]

q5-mithal-monitor/app.py

@@ -0,0 +1,51 @@
+from flask import Flask, send_file, jsonify
+import threading
+import time
+import json
+import os
+
+from monitor import collect_metrics
+
+app = Flask(__name__)
+
+
+def monitor_loop():
+    while True:
+        try:
+            collect_metrics()
+            print("Metrics updated.")
+        except Exception as e:
+            print("Monitor error:", e)
+
+        time.sleep(60)
+
+
+@app.route("/")
+def dashboard():
+    return send_file("dashboard.html")
+
+
+@app.route("/metrics.json")
+def metrics():
+    if os.path.exists("metrics.json"):
+        with open("metrics.json") as f:
+            return jsonify(json.load(f))
+
+    return jsonify({
+        "uptime": 0,
+        "last_check": {},
+        "history": [],
+        "latency_chart": []
+    })
+
+
+if __name__ == "__main__":
+    # Run one check immediately
+    collect_metrics()
+
+    # Start background monitoring
+    thread = threading.Thread(target=monitor_loop, daemon=True)
+    thread.start()
+
+    # Start Flask
+    app.run(host="0.0.0.0", port=8080)

q5-mithal-monitor/dashboard.html

@@ -0,0 +1,204 @@
+<!DOCTYPE html>
+<html lang="en">
+<head>
+<meta charset="UTF-8">
+<meta name="viewport" content="width=device-width, initial-scale=1.0">
+<title>Mithal Monitoring Dashboard</title>
+
+<script src="https://cdn.jsdelivr.net/npm/chart.js"></script>
+
+<style>
+body{
+    font-family:Arial,sans-serif;
+    background:#f4f4f4;
+    margin:20px;
+}
+h1{
+    text-align:center;
+}
+.cards{
+    display:grid;
+    grid-template-columns:repeat(auto-fit,minmax(220px,1fr));
+    gap:20px;
+    margin:20px 0;
+}
+.card{
+    background:white;
+    padding:20px;
+    border-radius:10px;
+    box-shadow:0 2px 6px rgba(0,0,0,.2);
+}
+.value{
+    font-size:28px;
+    color:#007bff;
+    font-weight:bold;
+}
+table{
+    width:100%;
+    border-collapse:collapse;
+    margin-top:30px;
+    background:white;
+}
+th,td{
+    border:1px solid #ddd;
+    padding:10px;
+    text-align:center;
+}
+th{
+    background:#007bff;
+    color:white;
+}
+canvas{
+    margin-top:30px;
+    background:white;
+    padding:15px;
+    border-radius:10px;
+}
+.good{
+    color:green;
+    font-weight:bold;
+}
+.bad{
+    color:red;
+    font-weight:bold;
+}
+</style>
+
+</head>
+<body>
+
+<h1>Mithal Monitoring Dashboard</h1>
+
+<div class="cards">
+
+<div class="card">
+<h3>Uptime</h3>
+<div id="uptime" class="value">--</div>
+</div>
+
+<div class="card">
+<h3>Latency</h3>
+<div id="latency" class="value">--</div>
+</div>
+
+<div class="card">
+<h3>DNS</h3>
+<div id="dns" class="value">--</div>
+</div>
+
+<div class="card">
+<h3>Search</h3>
+<div id="search" class="value">--</div>
+</div>
+
+<div class="card">
+<h3>SSL Expiry</h3>
+<div id="ssl" class="value" style="font-size:18px">--</div>
+</div>
+
+<div class="card">
+<h3>Days Left</h3>
+<div id="days" class="value">--</div>
+</div>
+
+</div>
+
+<canvas id="chart"></canvas>
+
+<table>
+
+<thead>
+
+<tr>
+
+<th>Time</th>
+<th>Status</th>
+<th>Latency</th>
+<th>DNS</th>
+<th>Search</th>
+<th>SSL Days</th>
+
+</tr>
+
+</thead>
+
+<tbody id="history"></tbody>
+
+</table>
+
+<script>
+
+let chart=null;
+
+async function loadMetrics(){
+
+    const response=await fetch("/metrics.json");
+
+    const data=await response.json();
+
+    document.getElementById("uptime").innerHTML=data.uptime+" %";
+    document.getElementById("latency").innerHTML=data.last_check.latency+" ms";
+    document.getElementById("dns").innerHTML=data.last_check.dns+" ms";
+    document.getElementById("search").innerHTML=data.last_check.search_latency+" ms";
+    document.getElementById("ssl").innerHTML=data.last_check.ssl_expiry;
+    document.getElementById("days").innerHTML=data.last_check.ssl_days_left;
+
+    const tbody=document.getElementById("history");
+
+    tbody.innerHTML="";
+
+    [...data.history].reverse().forEach(item=>{
+
+        tbody.innerHTML+=`
+        <tr>
+            <td>${item.time}</td>
+            <td class="${item.status==200?'good':'bad'}">${item.status}</td>
+            <td>${item.latency}</td>
+            <td>${item.dns}</td>
+            <td>${item.search_latency}</td>
+            <td>${item.ssl_days_left}</td>
+        </tr>`;
+    });
+
+    const labels=data.latency_chart.map((_,i)=>i+1);
+
+    if(chart){
+        chart.destroy();
+    }
+
+    chart=new Chart(document.getElementById("chart"),{
+
+        type:"line",
+
+        data:{
+
+            labels:labels,
+
+            datasets:[{
+
+                label:"Latency (ms)",
+
+                data:data.latency_chart,
+
+                borderColor:"#007bff",
+
+                fill:false,
+
+                tension:.3
+
+            }]
+
+        }
+
+    });
+
+}
+
+loadMetrics();
+
+setInterval(loadMetrics,60000);
+
+</script>
+
+</body>
+</html>

q5-mithal-monitor/monitor.py

@@ -0,0 +1,92 @@
+import requests
+import socket
+import ssl
+import time
+import json
+import os
+from datetime import datetime
+from urllib.parse import urlparse
+
+URL = "https://mithal.space"
+OUTPUT_FILE = "metrics.json"
+
+
+def collect_metrics():
+    hostname = urlparse(URL).hostname
+
+    # HTTP latency & uptime
+    start = time.time()
+    try:
+        response = requests.get(URL, timeout=10)
+        latency = round((time.time() - start) * 1000, 2)
+        status = response.status_code
+    except Exception:
+        latency = -1
+        status = 0
+
+    # DNS lookup
+    start = time.time()
+    try:
+        socket.gethostbyname(hostname)
+        dns = round((time.time() - start) * 1000, 2)
+    except Exception:
+        dns = -1
+
+    # SSL expiry
+    ssl_expiry = "Unavailable"
+    ssl_days = -1
+    try:
+        context = ssl.create_default_context()
+        with socket.create_connection((hostname, 443), timeout=10) as sock:
+            with context.wrap_socket(sock, server_hostname=hostname) as ssock:
+                cert = ssock.getpeercert()
+
+        expiry = datetime.strptime(cert["notAfter"], "%b %d %H:%M:%S %Y %Z")
+        ssl_expiry = expiry.strftime("%Y-%m-%d")
+        ssl_days = (expiry - datetime.utcnow()).days
+    except Exception:
+        pass
+
+    # Search response
+    start = time.time()
+    try:
+        requests.get(URL, timeout=10)
+        search_latency = round((time.time() - start) * 1000, 2)
+    except Exception:
+        search_latency = -1
+
+    if os.path.exists(OUTPUT_FILE):
+        with open(OUTPUT_FILE) as f:
+            data = json.load(f)
+    else:
+        data = {"history": []}
+
+    history = data.get("history", [])
+
+    history.append({
+        "time": datetime.now().strftime("%Y-%m-%d %H:%M:%S"),
+        "status": status,
+        "latency": latency,
+        "dns": dns,
+        "ssl_expiry": ssl_expiry,
+        "ssl_days_left": ssl_days,
+        "search_latency": search_latency
+    })
+
+    history = history[-1440:]
+
+    uptime = round(
+        len([x for x in history if x["status"] == 200]) /
+        len(history) * 100,
+        2
+    )
+
+    output = {
+        "uptime": uptime,
+        "last_check": history[-1],
+        "history": history[-10:],
+        "latency_chart": [x["latency"] for x in history[-60:]]
+    }
+
+    with open(OUTPUT_FILE, "w") as f:
+        json.dump(output, f, indent=4)

q5-mithal-monitor/requirements.txt

@@ -0,0 +1,2 @@
+Flask
+requests