ghaymah-exam-MohamedSaleh-SRE/README.md

Ghaymah SRE Exam Submission

This repository contains a completed set of SRE deliverables for the Ghaymah exam. It includes a containerized Flask API with health monitoring, an incident postmortem, a CI/CD workflow, scalability calculations, and an independent monitoring dashboard for https://mithal.space.

Contents

Area	Location	Summary
Q1 — Deployment & monitoring	q1-deploy-monitor/	Docker image, health-check script, and live API dashboard
Q2 — Postmortem	q2-postmortem/postmortem-report.md	SEV-1 OOMKilled incident analysis and prevention plan
Q3 — CI/CD	q3-cicd/	GitHub Actions workflow and environment/deployment guidance
Q4 — Scalability	q4-scalability/	Capacity calculation, cold-start strategy, and architecture diagram
Q5 — Mithal monitor	q5-mithal-monitor/	Website monitor, metric history, and monitoring dashboard
Community profile	common-qabilah/qabilah-profile.md	Qabilah profile link

Prerequisites

• Python 3.12+ for the main API and monitor
• Docker (optional, for containerized runs)
• curl to use the continuous health-check script
• Network access to https://mithal.space when running the Mithal monitor

Q1 — API deployment and health monitoring

The root Flask application provides a lightweight service health endpoint and a browser dashboard. It tracks its process uptime and the number of requests handled since startup.

App URL on Ghaymah Systems: https://ghaymah-sre-api-320ed50fd626.hosted.ghaymah.systems

Run locally

python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt
python app.py

The service listens on 0.0.0.0:5000.

URL	Purpose
http://localhost:5000/	Service landing page
http://localhost:5000/health	JSON health status, uptime, and request count
http://localhost:5000/dashboard	Browser dashboard polling the health endpoint every 30 seconds

Example health response:

{
  "requests": 1,
  "status": "healthy",
  "uptime": 0.02
}

Run with Docker

Build from the repository root so the Dockerfile can copy the shared application files:

docker build -f q1-deploy-monitor/Dockerfile -t ghaymah-sre-api .
docker run --rm -p 5000:5000 ghaymah-sre-api

Continuous health check

With the API running on port 5000, run the following in a second terminal:

bash q1-deploy-monitor/health-check.sh

The script requests /health every 30 seconds and prints the UTC-independent local timestamp, availability state, and request latency. Stop it with Ctrl+C.

Q2 — OOMKilled postmortem

The postmortem report documents a 45-minute SEV-1 outage caused by repeated container OOM kills under increased load. It covers:

• Incident impact and a detailed recovery timeline
• Root cause and contributing factors
• Immediate remediation and prevention measures
• HPA policy recommendations and early-detection metrics/alerts

Q3 — CI/CD

q3-cicd/workflow.yml defines a GitHub Actions pipeline triggered by pushes to main:

Build image → Authenticate to Ghaymah registry → Push image → Deploy staging → Deploy production

The build uses q1-deploy-monitor/Dockerfile and tags the image with the Git commit SHA. The registry steps require these repository secrets:

Secret	Purpose
GHAYMAH_USERNAME	Ghaymah registry account name
GHAYMAH_TOKEN	Ghaymah registry authentication token

Staging is represented by the staging GitHub environment. Production uses the production environment; configure its environment protection rules to require manual approval before the production job starts.

See the CI/CD documentation for the staging/production distinction and Ghaymah CLI commands.

Q4 — Scalability and load balancing

The capacity plan assumes 15,000 requests per second, a nominal capacity of 500 requests per second per container, and a 30% safety margin:

Effective capacity/container = 500 × 70% = 350 requests/second
Required containers          = ceil(15,000 ÷ 350) = 43

The plan recommends warm containers, autoscaling at CPU above 70%, pre-pulled images, and readiness/liveness checks before load-balancer registration. It also describes suitable persistent-workload use cases for Ghaymah Block Storage.

For the complete rationale, see the calculations.

Q5 — Mithal monitoring dashboard

This is a separate Flask service that monitors https://mithal.space every 60 seconds. Each collection records:

App URL on Ghaymah Systems: https://mithal-dashboard-e37761be7d74.hosted.ghaymah.systems

• HTTP status and response latency
• DNS lookup latency
• TLS certificate expiration date and days remaining
• A second endpoint-request latency, shown as “Search” in the dashboard
• Uptime percentage based on successful (200) checks

Metric samples are retained in metrics.json: the last 1,440 samples are kept internally, while the API returns the latest 10 history entries and latest 60 latency values for the chart. The data file is generated at runtime and is not included in the repository.

Run locally

cd q5-mithal-monitor
python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt
python app.py

Open http://localhost:8080/ for the dashboard, or retrieve raw metrics from http://localhost:8080/metrics.json.

Run with Docker

Build using the monitor directory as the build context:

docker build -t mithal-monitor q5-mithal-monitor
docker run --rm -p 8080:8080 mithal-monitor

The dashboard loads Chart.js from a public CDN, so it needs browser internet access to render the latency chart.

Project layout

.
├── app.py                         # Main Flask API
├── requirements.txt               # Main API dependencies
├── q1-deploy-monitor/             # API Dockerfile, dashboard, health checker
├── q2-postmortem/                 # Incident postmortem
├── q3-cicd/                       # GitHub Actions workflow and CI/CD notes
├── q4-scalability/                # Capacity analysis and architecture image
├── q5-mithal-monitor/             # Independent external-site monitor
└── common-qabilah/                # Community profile

Notes and limitations

• Both Flask applications use the development server, which is appropriate for this exercise. A production deployment should use a production WSGI server such as Gunicorn (already listed in the root requirements) behind a reverse proxy/load balancer.
• The main API’s request counter and uptime are in-memory and reset whenever the application restarts.
• The Mithal monitor writes metrics to its local container filesystem. Mount persistent storage if history must survive container replacement.
• The deployment jobs currently contain placeholder deployment commands; connect them to the target Ghaymah environment before using the workflow for a live release.