# Ghaymah Test — Step-by-Step Checklist

Work through this top to bottom. Each step is tagged with who does it:
**[YOU]** = manual work · **[OPUS]** = prompt Opus 5 · **[SOL]** = prompt Sol/Codex
Steps inside the same phase that have different tags can run at the same time.

---

## PHASE 0 — Setup & Research (Day 1 morning, ~2–3h)

- [ ] **0.1 [YOU]** Create a Git repo `ghaymah-test` with this folder structure:
  ```
  q1-deploy-monitor/{app,monitor,dashboard}
  q2-postmortem/
  q3-cicd/
  q4-scalability/
  q5-mithal-dashboard/{collector,data,dashboard}
  README.md
  ```
- [ ] **0.2 [YOU]** Create/log in to your **ghaymah.systems** account.
- [ ] **0.3 [YOU]** Explore and take notes + screenshots on:
  - [ ] Container platform (how to create a service, set port, get public URL)
  - [ ] Container Registry (registry URL, how to log in/push)
  - [ ] ghaymah CLI (install command, auth command, deploy command)
  - [ ] Monitoring tools (metrics, graphs, alert rules)
  - [ ] Block Storage (how volumes attach, limits)
  > These notes go into every agent prompt — Q2/Q3/Q4 must cite real Ghaymah features.
- [ ] **0.4 [YOU]** Open **mithal.space** in a browser, run a search, and copy the search URL pattern (e.g. `/search?q=test`). Save it for step 4.1.
- [ ] **0.5 [YOU]** Decide stack: **Python + FastAPI** (assumed below).

---

## PHASE 1 — Dispatch both agents (Day 1 afternoon)

Send these two prompts at the same time — the tracks don't share files.

- [ ] **1.1 [OPUS]** Prompt Opus with: full Q1 text + your platform notes + repo layout. Ask for:
  - [ ] `q1-deploy-monitor/app/main.py` — FastAPI with `GET /`, `GET /health` (returns `{"status":"ok","uptime_s":...,"timestamp":...}`), `GET /metrics` (in-memory request counter via middleware)
  - [ ] `app/requirements.txt` (fastapi, uvicorn)
  - [ ] `app/Dockerfile` — `python:3.12-slim`, non-root user, layer-cached COPY order, `EXPOSE 8080`, `HEALTHCHECK` hitting `/health`, plus `.dockerignore`
  - [ ] `monitor/monitor.py` — loop every 30s: GET `$APP_URL/health` (5s timeout), append `{"ts","status","code","latency_ms","requests"}` to `monitor/data/checks.json`; "down" on exception; alert print after 3 consecutive fails
  - [ ] `dashboard/index.html` — single file, no build step: green/red status badge, latency line chart (Chart.js CDN), request-count tile, fetches `checks.json`, auto-refresh 30s
- [ ] **1.2 [SOL]** Prompt Sol with: full Q2 text + your monitoring-tool notes. Ask for `q2-postmortem/POSTMORTEM.md` containing:
  - [ ] Summary (45-min outage, OOMKilled exit 137, impact, severity)
  - [ ] Minute-by-minute timeline table (memory creep → OOMKill → crash-loop → alert → limit raise + rollback → recovery)
  - [ ] Root cause via 5-Whys (memory leak in release X + undersized limit + no memory alerting/CI check)
  - [ ] Recommendations table (owner + priority)
  - [ ] Auto-scaling policy: scale out at mem>70% or CPU>65% for 2 min, min 2 replicas, scale-in cooldown 5–10 min — **and** an explicit note that HPA alone doesn't fix a leak
  - [ ] Early-detection section citing real Ghaymah monitoring features; every platform claim marked `<!-- VERIFY -->`

---

## PHASE 2 — Q1 goes live (Day 2 morning)

- [ ] **2.1 [YOU]** Review Opus's Q1 output. Build & test locally:
  ```
  docker build -t ghaymah-api ./q1-deploy-monitor/app
  docker run -p 8080:8080 ghaymah-api
  ```
  - [ ] `http://localhost:8080/health` returns 200 JSON
  - [ ] `/metrics` counter increments
- [ ] **2.2 [YOU]** Push image to ghaymah Container Registry (`docker login` → `docker tag` → `docker push`).
- [ ] **2.3 [YOU]** Create the container service on ghaymah.systems (port 8080), note the **public URL**, screenshot the running service + `/health` response.
- [ ] **2.4 [YOU]** Start `monitor.py` with `APP_URL=<public URL>` and **leave it running** so real data accumulates before submission.
- [ ] **2.5 [YOU]** Open the Q1 dashboard against the growing `checks.json` — verify badge, chart, counter all render.
- [ ] **2.6 [OPUS]** Send any deployment fixes back to Opus, then immediately give it Q5 (step 4.1) — don't wait.

---

## PHASE 3 — CI/CD (Day 2 morning, parallel with Phase 2)

- [ ] **3.1 [SOL]** Prompt Sol with: full Q3 text + your CLI/registry notes. Ask for:
  - [ ] `.github/workflows/deploy.yml` — trigger on push to `main` + `workflow_dispatch`; jobs:
    - `build-push`: checkout → `docker/login-action` (secrets `GHAYMAH_REGISTRY_USER`/`TOKEN`) → `docker/build-push-action` tagging `:${{ github.sha }}` and `:latest`
    - `deploy-staging` (needs build-push): deploy via ghaymah CLI
    - `deploy-production` (needs staging): bound to GitHub Environment `production` (this is the manual-approval gate)
  - [ ] `q3-cicd/CICD.md` — staging vs production section (data, scale, secrets, access, alerting, cadence + promotion flow diagram) and ghaymah CLI integration section (install, `auth login` with `GHAYMAH_API_TOKEN` secret, deploy command); placeholders marked `<!-- VERIFY -->`
- [ ] **3.2 [YOU]** Push repo to GitHub. In repo settings:
  - [ ] Add secrets: `GHAYMAH_REGISTRY_USER`, `GHAYMAH_REGISTRY_TOKEN`, `GHAYMAH_API_TOKEN`
  - [ ] Settings → Environments → create `production` → add yourself as **required reviewer**
- [ ] **3.3 [YOU]** Trigger the workflow, let it pause at the production gate, **screenshot the approval prompt**, then approve.

---

## PHASE 4 — Q5 build & deploy (Day 2 afternoon)

- [ ] **4.1 [OPUS]** Prompt Opus with: full Q5 text + the mithal.space search URL from 0.4. Ask for:
  - [ ] `collector/collect.py` — every 60s (plus `--once` flag) measure:
    - Latency: timed `GET https://mithal.space`
    - Uptime: up if status 200–399
    - SSL: `ssl`/`socket` cert `notAfter` → days remaining
    - DNS: timed `socket.getaddrinfo`
    - Search: timed request to the search URL (configurable)
    - Append `{"ts","up","code","latency_ms","dns_ms","ssl_days_left","search_ms"}` to `data/metrics.json`, prune >48h
  - [ ] `dashboard/index.html` — uptime % tile (24h), latency line chart (last hour, Chart.js), SSL card color-coded (green>30d / yellow 8–30 / red≤7), last-10-checks table, auto-refresh 60s; same visual style as Q1
  - [ ] `Dockerfile` — one container serving the dashboard statically **and** running the collector in the background (shell entrypoint launching both), collector writes into the served `data/` dir
- [ ] **4.2 [YOU]** Test locally (`--once` first, then the loop; open the dashboard). Fix issues via Opus.
- [ ] **4.3 [YOU]** Push image + deploy to ghaymah exactly like 2.2–2.3. Screenshot the live dashboard URL.
- [ ] **4.4 [YOU]** Leave it running so the 24h uptime % and hourly chart fill with real data.

---

## PHASE 5 — Q4 scalability doc (Day 2 afternoon, parallel with Phase 4)

- [ ] **5.1 [SOL]** Prompt Sol with: full Q4 text + your Block Storage notes. Ask for `q4-scalability/SCALABILITY.md`:
  - [ ] Mermaid architecture diagram: DNS → CDN → L7 load balancer → stateless API fleet (auto-scaled) → Redis cache → DB primary+replicas on Block Storage; monitoring on the side; annotate the 15k req/s path
  - [ ] Container math shown explicitly: `15,000 ÷ 500 = 30 → ×1.30 = 39 containers`, plus notes on rounding up and N+1 for rolling deploys
  - [ ] Cold-start strategy: warm pool / min replicas, slim pre-pulled images, readiness-probe gating, predictive scaling before peaks, step scale-out policies
  - [ ] Block Storage section: ephemeral containers vs persistent volumes, one-writer semantics, snapshots/backups, IOPS; API tier stays diskless — `<!-- VERIFY -->` markers on platform specifics

---

## PHASE 6 — Verification & polish (Day 3 morning)

- [ ] **6.1 [YOU]** Search the whole repo for `<!-- VERIFY -->` and resolve every one against real Ghaymah docs (fix CLI commands, registry URL, monitoring feature names, Block Storage details).
- [ ] **6.2 [YOU]** Insert real values everywhere: live app URL (Q1), registry image name (Q3), live dashboard URL (Q5).
- [ ] **6.3 [YOU]** Add all screenshots to the repo: Q1 service + `/health`, Q3 approval gate, Q5 live dashboard.
- [ ] **6.4 [SOL]** Send Sol any corrections needed in Q2/Q3/Q4 docs from your verification pass.
- [ ] **6.5 [OPUS]** Ask Opus for final dashboard polish now that real data exists (empty-state handling, chart scaling).

---

## PHASE 7 — Final review & submit (Day 3 afternoon)

- [ ] **7.1 [YOU]** Rubric pass — check every numbered sub-requirement one by one:
  - [ ] Q1: Dockerfile · deployed on ghaymah · `/health` · 30s monitor script · dashboard (status, latency, request count)
  - [ ] Q2: postmortem (summary, timeline, root cause, recommendations) · auto-scaling policy · early-detection with Ghaymah tools
  - [ ] Q3: build+push workflow · manual approval before production · staging vs production explained · CLI integration documented
  - [ ] Q4: architecture diagram · container count (39) with math · cold-start strategy · Block Storage explanation
  - [ ] Q5: collector (latency, uptime, SSL, DNS, search) every minute · CSV/JSON storage · dashboard (uptime %, latency chart, SSL, last 10 checks) · deployed on ghaymah
- [ ] **7.2 [YOU]** Write `README.md` as an index: one section per question, links to every file, live URLs, screenshots.
- [ ] **7.3 [YOU]** Confirm both dashboards show real accumulated data (not empty charts).
- [ ] **7.4 [YOU]** Final commit + push. Submit.