Edit README
هذا الالتزام موجود في:
root
171
README.md
171
README.md
@@ -0,0 +1,171 @@
|
||||
# Ghaymah SRE Exam Submission
|
||||
|
||||
This repository contains a completed set of SRE deliverables for the Ghaymah exam. It includes a containerized Flask API with health monitoring, an incident postmortem, a CI/CD workflow, scalability calculations, and an independent monitoring dashboard for `https://mithal.space`.
|
||||
|
||||
## Contents
|
||||
|
||||
| Area | Location | Summary |
|
||||
| --- | --- | --- |
|
||||
| Q1 — Deployment & monitoring | [`q1-deploy-monitor/`](q1-deploy-monitor/) | Docker image, health-check script, and live API dashboard |
|
||||
| Q2 — Postmortem | [`q2-postmortem/postmortem-report.md`](q2-postmortem/postmortem-report.md) | SEV-1 OOMKilled incident analysis and prevention plan |
|
||||
| Q3 — CI/CD | [`q3-cicd/`](q3-cicd/) | GitHub Actions workflow and environment/deployment guidance |
|
||||
| Q4 — Scalability | [`q4-scalability/`](q4-scalability/) | Capacity calculation, cold-start strategy, and architecture diagram |
|
||||
| Q5 — Mithal monitor | [`q5-mithal-monitor/`](q5-mithal-monitor/) | Website monitor, metric history, and monitoring dashboard |
|
||||
| Community profile | [`common-qabilah/qabilah-profile.md`](common-qabilah/qabilah-profile.md) | Qabilah profile link |
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- Python 3.12+ for the main API and monitor
|
||||
- Docker (optional, for containerized runs)
|
||||
- `curl` to use the continuous health-check script
|
||||
- Network access to `https://mithal.space` when running the Mithal monitor
|
||||
|
||||
## Q1 — API deployment and health monitoring
|
||||
|
||||
The root Flask application provides a lightweight service health endpoint and a browser dashboard. It tracks its process uptime and the number of requests handled since startup.
|
||||
|
||||
### Run locally
|
||||
|
||||
```bash
|
||||
python3 -m venv .venv
|
||||
source .venv/bin/activate
|
||||
pip install -r requirements.txt
|
||||
python app.py
|
||||
```
|
||||
|
||||
The service listens on `0.0.0.0:5000`.
|
||||
|
||||
| URL | Purpose |
|
||||
| --- | --- |
|
||||
| `http://localhost:5000/` | Service landing page |
|
||||
| `http://localhost:5000/health` | JSON health status, uptime, and request count |
|
||||
| `http://localhost:5000/dashboard` | Browser dashboard polling the health endpoint every 30 seconds |
|
||||
|
||||
Example health response:
|
||||
|
||||
```json
|
||||
{
|
||||
"requests": 1,
|
||||
"status": "healthy",
|
||||
"uptime": 0.02
|
||||
}
|
||||
```
|
||||
|
||||
### Run with Docker
|
||||
|
||||
Build from the repository root so the Dockerfile can copy the shared application files:
|
||||
|
||||
```bash
|
||||
docker build -f q1-deploy-monitor/Dockerfile -t ghaymah-sre-api .
|
||||
docker run --rm -p 5000:5000 ghaymah-sre-api
|
||||
```
|
||||
|
||||
### Continuous health check
|
||||
|
||||
With the API running on port 5000, run the following in a second terminal:
|
||||
|
||||
```bash
|
||||
bash q1-deploy-monitor/health-check.sh
|
||||
```
|
||||
|
||||
The script requests `/health` every 30 seconds and prints the UTC-independent local timestamp, availability state, and request latency. Stop it with `Ctrl+C`.
|
||||
|
||||
## Q2 — OOMKilled postmortem
|
||||
|
||||
The [postmortem report](q2-postmortem/postmortem-report.md) documents a 45-minute SEV-1 outage caused by repeated container OOM kills under increased load. It covers:
|
||||
|
||||
- Incident impact and a detailed recovery timeline
|
||||
- Root cause and contributing factors
|
||||
- Immediate remediation and prevention measures
|
||||
- HPA policy recommendations and early-detection metrics/alerts
|
||||
|
||||
## Q3 — CI/CD
|
||||
|
||||
[`q3-cicd/workflow.yml`](q3-cicd/workflow.yml) defines a GitHub Actions pipeline triggered by pushes to `main`:
|
||||
|
||||
```text
|
||||
Build image → Authenticate to Ghaymah registry → Push image → Deploy staging → Deploy production
|
||||
```
|
||||
|
||||
The build uses `q1-deploy-monitor/Dockerfile` and tags the image with the Git commit SHA. The registry steps require these repository secrets:
|
||||
|
||||
| Secret | Purpose |
|
||||
| --- | --- |
|
||||
| `GHAYMAH_USERNAME` | Ghaymah registry account name |
|
||||
| `GHAYMAH_TOKEN` | Ghaymah registry authentication token |
|
||||
|
||||
Staging is represented by the `staging` GitHub environment. Production uses the `production` environment; configure its environment protection rules to require manual approval before the production job starts.
|
||||
|
||||
See the [CI/CD documentation](q3-cicd/README.md) for the staging/production distinction and Ghaymah CLI commands.
|
||||
|
||||
## Q4 — Scalability and load balancing
|
||||
|
||||
The capacity plan assumes 15,000 requests per second, a nominal capacity of 500 requests per second per container, and a 30% safety margin:
|
||||
|
||||
```text
|
||||
Effective capacity/container = 500 × 70% = 350 requests/second
|
||||
Required containers = ceil(15,000 ÷ 350) = 43
|
||||
```
|
||||
|
||||
The plan recommends warm containers, autoscaling at CPU above 70%, pre-pulled images, and readiness/liveness checks before load-balancer registration. It also describes suitable persistent-workload use cases for Ghaymah Block Storage.
|
||||
|
||||
|
||||
|
||||
For the complete rationale, see [the calculations](q4-scalability/calculations.md).
|
||||
|
||||
## Q5 — Mithal monitoring dashboard
|
||||
|
||||
This is a separate Flask service that monitors `https://mithal.space` every 60 seconds. Each collection records:
|
||||
|
||||
- HTTP status and response latency
|
||||
- DNS lookup latency
|
||||
- TLS certificate expiration date and days remaining
|
||||
- A second endpoint-request latency, shown as “Search” in the dashboard
|
||||
- Uptime percentage based on successful (`200`) checks
|
||||
|
||||
Metric samples are retained in `metrics.json`: the last 1,440 samples are kept internally, while the API returns the latest 10 history entries and latest 60 latency values for the chart. The data file is generated at runtime and is not included in the repository.
|
||||
|
||||
### Run locally
|
||||
|
||||
```bash
|
||||
cd q5-mithal-monitor
|
||||
python3 -m venv .venv
|
||||
source .venv/bin/activate
|
||||
pip install -r requirements.txt
|
||||
python app.py
|
||||
```
|
||||
|
||||
Open `http://localhost:8080/` for the dashboard, or retrieve raw metrics from `http://localhost:8080/metrics.json`.
|
||||
|
||||
### Run with Docker
|
||||
|
||||
Build using the monitor directory as the build context:
|
||||
|
||||
```bash
|
||||
docker build -t mithal-monitor q5-mithal-monitor
|
||||
docker run --rm -p 8080:8080 mithal-monitor
|
||||
```
|
||||
|
||||
> The dashboard loads Chart.js from a public CDN, so it needs browser internet access to render the latency chart.
|
||||
|
||||
## Project layout
|
||||
|
||||
```text
|
||||
.
|
||||
├── app.py # Main Flask API
|
||||
├── requirements.txt # Main API dependencies
|
||||
├── q1-deploy-monitor/ # API Dockerfile, dashboard, health checker
|
||||
├── q2-postmortem/ # Incident postmortem
|
||||
├── q3-cicd/ # GitHub Actions workflow and CI/CD notes
|
||||
├── q4-scalability/ # Capacity analysis and architecture image
|
||||
├── q5-mithal-monitor/ # Independent external-site monitor
|
||||
└── common-qabilah/ # Community profile
|
||||
```
|
||||
|
||||
## Notes and limitations
|
||||
|
||||
- Both Flask applications use the development server, which is appropriate for this exercise. A production deployment should use a production WSGI server such as Gunicorn (already listed in the root requirements) behind a reverse proxy/load balancer.
|
||||
- The main API’s request counter and uptime are in-memory and reset whenever the application restarts.
|
||||
- The Mithal monitor writes metrics to its local container filesystem. Mount persistent storage if history must survive container replacement.
|
||||
- The deployment jobs currently contain placeholder deployment commands; connect them to the target Ghaymah environment before using the workflow for a live release.
|
||||
|
||||
|
||||
المرجع في مشكلة جديدة
حظر مستخدم