# Ghaymah SRE Exam Submission

This repository contains a completed set of SRE deliverables for the Ghaymah exam. It includes a containerized Flask API with health monitoring, an incident postmortem, a CI/CD workflow, scalability calculations, and an independent monitoring dashboard for `https://mithal.space`.

## Contents

| Area | Location | Summary |
| --- | --- | --- |
| Q1 — Deployment & monitoring | [`q1-deploy-monitor/`](q1-deploy-monitor/) | Docker image, health-check script, and live API dashboard |
| Q2 — Postmortem | [`q2-postmortem/postmortem-report.md`](q2-postmortem/postmortem-report.md) | SEV-1 OOMKilled incident analysis and prevention plan |
| Q3 — CI/CD | [`q3-cicd/`](q3-cicd/) | GitHub Actions workflow and environment/deployment guidance |
| Q4 — Scalability | [`q4-scalability/`](q4-scalability/) | Capacity calculation, cold-start strategy, and architecture diagram |
| Q5 — Mithal monitor | [`q5-mithal-monitor/`](q5-mithal-monitor/) | Website monitor, metric history, and monitoring dashboard |
| Community profile | [`common-qabilah/qabilah-profile.md`](common-qabilah/qabilah-profile.md) | Qabilah profile link |

## Prerequisites

- Python 3.12+ for the main API and monitor
- Docker (optional, for containerized runs)
- `curl` to use the continuous health-check script
- Network access to `https://mithal.space` when running the Mithal monitor

## Q1 — API deployment and health monitoring

The root Flask application provides a lightweight service health endpoint and a browser dashboard. It tracks its process uptime and the number of requests handled since startup.

**App URL on Ghaymah Systems: [https://ghaymah-sre-api-320ed50fd626.hosted.ghaymah.systems](https://ghaymah-sre-api-320ed50fd626.hosted.ghaymah.systems)**

### Run locally

```bash
python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt
python app.py
```

The service listens on `0.0.0.0:5000`.

| URL | Purpose |
| --- | --- |
| `http://localhost:5000/` | Service landing page |
| `http://localhost:5000/health` | JSON health status, uptime, and request count |
| `http://localhost:5000/dashboard` | Browser dashboard polling the health endpoint every 30 seconds |

Example health response:

```json
{
  "requests": 1,
  "status": "healthy",
  "uptime": 0.02
}
```

### Run with Docker

Build from the repository root so the Dockerfile can copy the shared application files:

```bash
docker build -f q1-deploy-monitor/Dockerfile -t ghaymah-sre-api .
docker run --rm -p 5000:5000 ghaymah-sre-api
```

### Continuous health check

With the API running on port 5000, run the following in a second terminal:

```bash
bash q1-deploy-monitor/health-check.sh
```

The script requests `/health` every 30 seconds and prints the UTC-independent local timestamp, availability state, and request latency. Stop it with `Ctrl+C`.

## Q2 — OOMKilled postmortem

The [postmortem report](q2-postmortem/postmortem-report.md) documents a 45-minute SEV-1 outage caused by repeated container OOM kills under increased load. It covers:

- Incident impact and a detailed recovery timeline
- Root cause and contributing factors
- Immediate remediation and prevention measures
- HPA policy recommendations and early-detection metrics/alerts

## Q3 — CI/CD

[`q3-cicd/workflow.yml`](q3-cicd/workflow.yml) defines a GitHub Actions pipeline triggered by pushes to `main`:

```text
Build image → Authenticate to Ghaymah registry → Push image → Deploy staging → Deploy production
```

The build uses `q1-deploy-monitor/Dockerfile` and tags the image with the Git commit SHA. The registry steps require these repository secrets:

| Secret | Purpose |
| --- | --- |
| `GHAYMAH_USERNAME` | Ghaymah registry account name |
| `GHAYMAH_TOKEN` | Ghaymah registry authentication token |

Staging is represented by the `staging` GitHub environment. Production uses the `production` environment; configure its environment protection rules to require manual approval before the production job starts.

See the [CI/CD documentation](q3-cicd/README.md) for the staging/production distinction and Ghaymah CLI commands.

## Q4 — Scalability and load balancing

The capacity plan assumes 15,000 requests per second, a nominal capacity of 500 requests per second per container, and a 30% safety margin:

```text
Effective capacity/container = 500 × 70% = 350 requests/second
Required containers          = ceil(15,000 ÷ 350) = 43
```

The plan recommends warm containers, autoscaling at CPU above 70%, pre-pulled images, and readiness/liveness checks before load-balancer registration. It also describes suitable persistent-workload use cases for Ghaymah Block Storage.

![Scalability architecture](q4-scalability/architecture.png)

For the complete rationale, see [the calculations](q4-scalability/calculations.md).

## Q5 — Mithal monitoring dashboard

This is a separate Flask service that monitors `https://mithal.space` every 60 seconds. Each collection records:

**App URL on Ghaymah Systems: [https://mithal-dashboard-e37761be7d74.hosted.ghaymah.systems](https://mithal-dashboard-e37761be7d74.hosted.ghaymah.systems)**

- HTTP status and response latency
- DNS lookup latency
- TLS certificate expiration date and days remaining
- A second endpoint-request latency, shown as “Search” in the dashboard
- Uptime percentage based on successful (`200`) checks

Metric samples are retained in `metrics.json`: the last 1,440 samples are kept internally, while the API returns the latest 10 history entries and latest 60 latency values for the chart. The data file is generated at runtime and is not included in the repository.

### Run locally

```bash
cd q5-mithal-monitor
python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt
python app.py
```

Open `http://localhost:8080/` for the dashboard, or retrieve raw metrics from `http://localhost:8080/metrics.json`.

### Run with Docker

Build using the monitor directory as the build context:

```bash
docker build -t mithal-monitor q5-mithal-monitor
docker run --rm -p 8080:8080 mithal-monitor
```

> The dashboard loads Chart.js from a public CDN, so it needs browser internet access to render the latency chart.

## Project layout

```text
.
├── app.py                         # Main Flask API
├── requirements.txt               # Main API dependencies
├── q1-deploy-monitor/             # API Dockerfile, dashboard, health checker
├── q2-postmortem/                 # Incident postmortem
├── q3-cicd/                       # GitHub Actions workflow and CI/CD notes
├── q4-scalability/                # Capacity analysis and architecture image
├── q5-mithal-monitor/             # Independent external-site monitor
└── common-qabilah/                # Community profile
```

## Notes and limitations

- Both Flask applications use the development server, which is appropriate for this exercise. A production deployment should use a production WSGI server such as Gunicorn (already listed in the root requirements) behind a reverse proxy/load balancer.
- The main API’s request counter and uptime are in-memory and reset whenever the application restarts.
- The Mithal monitor writes metrics to its local container filesystem. Mount persistent storage if history must survive container replacement.
- The deployment jobs currently contain placeholder deployment commands; connect them to the target Ghaymah environment before using the workflow for a live release.