# Ghaymah Cloud Internship — SRE & SecOps Track

> Deployment, Monitoring, CI/CD, Scalability, and Observability on **ghaymah.systems**

**Author:** Mohamed Moustafa
**Track:** SRE & SecOps
**Platform:** [ghaymah.systems](https://ghaymah.systems)

---

## Project Overview

This repository contains all six tasks from the Ghaymah Cloud internship exam, covering the full lifecycle of cloud-native applications — from containerized deployment to post-incident analysis, CI/CD automation, scalability design, and real-time monitoring.

| Task | Topic | Score | Status |
|------|-------|-------|--------|
| Q1 | Deploy & Monitor on Ghaymah | 20/20 | Done |
| Q2 | Postmortem — OOMKilled Incident | 20/20 | Done |
| Q3 | CI/CD Pipeline with GitHub Actions | 20/20 | Done |
| Q4 | Scalability & Load Balancing | 20/20 | Done |
| Q5 | Mithal.space Monitoring Dashboard | 20/20 | Done |
| Q6 | Mortakaz Integration Proposals | 15/15 | Done |

---

## Repository Structure

```
ghaymah-exam-mohamed-sre/
├── README.md                          ← You are here
│
├── q1-deploy-monitor/
│   ├── Dockerfile                     ← Multi-stage Node.js build
│   ├── server.js                      ← Express API with /health endpoint
│   ├── package.json
│   ├── health-check.sh                ← Bash monitor (30s interval)
│   ├── ghaymah.json                   ← Ghaymah CLI config
│   ├── architecture.svg               ← Architecture diagram
│   └── public/
│       ├── dashboard.html             ← Monitoring dashboard
│       ├── dashboard.css
│       └── dashboard.js
│
├── q2-postmortem/
│   ├── postmortem-report.md           ← Full incident report
│   └── timeline.svg                   ← Visual incident timeline
│
├── q3-cicd/
│   ├── workflow.yml                   ← GitHub Actions workflow
│   ├── pipeline.svg                   ← CI/CD pipeline diagram
│   └── Readme.md                      ← Staging vs Production + CLI guide
│
├── q4-scalability/
│   ├── calculations.md                ← Container math & strategy
│   ├── architecture.svg               ← 15K req/s architecture
│   └── architecture.png               ← PNG version
│
├── q5-mithal-monitor/
│   ├── monitor.py                     ← Python monitoring script
│   ├── dashboard.html                 ← Real-time dashboard
│   ├── style.css                      ← Dashboard styles
│   ├── script.js                      ← Dashboard logic
│   ├── Dockerfile                     ← Docker image (Python + Nginx)
│   ├── nginx.conf                     ← Nginx config
│   ├── entrypoint.sh                  ← Container entrypoint
│   ├── requirements.txt               ← Python dependencies
│   ├── README.md                      ← Detailed documentation
│   └── screenshots/
│       ├── dashboard-preview.svg      ← Dashboard preview
│       └── monitor-preview.svg        ← Terminal preview
│
├── common-mortakaz/
│   ├── integration-1.md               ← Circle Panel proposal
│   └── integration-2.md               ← Bilya AI Assistant proposal
│
└── common-qabilah/
    └── qabilah-profile.txt            ← Qabilah profile link
```

---

## Q1 — Deploy & Monitor on Ghaymah

**Goal:** Containerize a Node.js API, deploy to ghaymah.systems, and monitor it.

### Architecture

![Q1 Architecture](q1-deploy-monitor/architecture.svg)

### What Was Built

- **Node.js API** (Express) with `/health` endpoint returning status and request count
- **Multi-stage Dockerfile** — builder stage for `npm ci`, production stage on Alpine
- **Health check script** (`health-check.sh`) — polls `/health` every 30 seconds, logs status
- **Monitoring dashboard** — live status, response time, total requests

### Key Files

| File | Purpose |
|------|---------|
| `Dockerfile` | Multi-stage build: `node:22` → `node:22-alpine` |
| `server.js` | Express server with `/health` endpoint |
| `health-check.sh` | Bash monitoring loop (30s interval) |
| `ghaymah.json` | Ghaymah CLI configuration |

### Deploy Commands

```bash
# Install Ghaymah CLI
curl -sSL https://cli.ghaymah.systems/install.sh | bash

# Authenticate
$HOME/ghaymah/bin/gy auth login --email "EMAIL" --password "PW"

# Deploy
$HOME/ghaymah/bin/gy resource app launch

# Run monitor
chmod +x health-check.sh
./health-check.sh
```

---

## Q2 — Postmortem: OOMKilled Incident

**Goal:** Document a 45-minute outage caused by repeated OOMKilled events.

### Incident Timeline

![Postmortem Timeline](q2-postmortem/timeline.svg)

### Summary

| Field | Detail |
|-------|--------|
| Duration | 45 minutes |
| Severity | High |
| Root Cause | Memory leak exceeded container limit |
| Impact | 100% request failure, 12+ restarts |

### Key Sections

- **Root Cause:** Application consumed more memory than the container's configured limit
- **Timeline:** Deploy → Memory spike → OOMKilled → Crash loop → Fix
- **Auto-Scaling Policy:** Scale-out at 80% memory, scale-in at 40%, min 2 / max 10 instances
- **Early Detection:** Prometheus + Grafana for memory metrics, alerts at 80% threshold

---

## Q3 — CI/CD Pipeline

**Goal:** Automate build, push, and deploy with manual approval gate.

### Pipeline

![CI/CD Pipeline](q3-cicd/pipeline.svg)

### Workflow Steps

1. **Trigger** — `git push` to `main` branch
2. **Build** — Docker image from Dockerfile
3. **Test** — `npm test` verification
4. **Push** — Image to Ghaymah Container Registry
5. **Approval** — Manual gate before production
6. **Deploy** — `gy resource app launch`

### Staging vs Production

| Feature | Staging | Production |
|---------|---------|------------|
| Users | Developers / QA | End Users |
| Data | Test Data | Production Data |
| Approval | Optional | Required |
| Stability | Medium | High |

### Ghaymah CLI Integration

```bash
# Install
curl -sSL https://cli.ghaymah.systems/install.sh | bash

# Login
gy auth login --email "EMAIL" --password "PW"

# Deploy
gy resource app launch

# Monitor
gy app logs
```

---

## Q4 — Scalability & Load Balancing

**Goal:** Design architecture for 15,000 req/s on Ghaymah Cloud.

### Architecture

![Scalability Architecture](q4-scalability/architecture.svg)

### Calculations

```
Traffic:       15,000 req/s
Per Container:  500 req/s
Base Need:      15,000 / 500 = 30 containers
Safety Margin:  30% → 30 × 1.3 = 39 containers
```

### Cold Start Strategy

- Keep **2–3 warm containers** ready
- Use **lightweight Docker images** (Alpine-based)
- Configure **health checks** before routing traffic
- Trigger auto-scaling at **70% CPU** or **80% memory**

### Block Storage for Stateful Workloads

Ghaymah Block Storage persists data across container restarts:

- **Databases:** PostgreSQL, MySQL, MongoDB
- **File Storage:** User uploads, recordings
- **Logs:** Persistent application logs
- **Backups:** Automated backup storage

---

## Q5 — Mithal.space Monitoring Dashboard

**Goal:** Build a production-quality monitoring solution for mithal.space.

### Dashboard Preview

![Dashboard](q5-mithal-monitor/screenshots/dashboard-preview.svg)

### Monitor Output

![Monitor](q5-mithal-monitor/screenshots/monitor-preview.svg)

### Features

| Feature | Implementation |
|---------|---------------|
| HTTP Latency | `requests` library, measures GET response time |
| Uptime | Status code check (200–399 = UP) |
| SSL Certificate | `ssl` module, checks expiry date |
| DNS Lookup | `socket.getaddrinfo()` timing |
| Search Response | Real `/search?q=` request measurement |
| Data Storage | `metrics.json` (1,440 records ≈ 24h) |
| Dashboard | HTML/CSS/JS with Chart.js |
| Dark Mode | Toggle with localStorage persistence |
| CSV Export | One-click metrics export |

### Quick Start

```bash
cd q5-mithal-monitor
pip install -r requirements.txt
python monitor.py --interval 60

# Dashboard
python3 -m http.server 8080
# Open http://localhost:8080/dashboard.html
```

### Docker

```bash
docker build -t mithal-monitor .
docker run -d -p 8080:80 mithal-monitor
```

---

## Q6 — Mortakaz Integration Proposals

**Goal:** Propose integrations between mortakaz.com products and Ghaymah Cloud.

### Product 1: Circle Panel

User research platform for Arabic-speaking product teams.

- **Ghaymah Integration:** Containerized microservices (Frontend, Backend, AI Processing)
- **Block Storage:** Interview recordings and research data
- **mithal.space:** Searchable documentation and resources

### Product 2: Bilya AI Assistant

AI-powered virtual assistant for automotive service centers.

- **Ghaymah Integration:** Microservices (Frontend, API, AI, Booking)
- **Block Storage:** Customer conversations and booking records
- **mithal.space:** FAQ and documentation discoverability

### Recommendation

**Bilya AI Assistant** is the strongest candidate — its architecture naturally benefits from containers, auto-scaling, persistent storage, and CI/CD pipelines.

---

## Qabilah Profile

[qabilah.com/profile/mohamed-moustafa20](https://qabilah.com/profile/mohamed-moustafa20/)

---

## Technology Stack

| Layer | Technology |
|-------|-----------|
| Runtime | Node.js 22, Python 3.12 |
| Framework | Express.js |
| Container | Docker (multi-stage) |
| Web Server | Nginx |
| Monitoring | Python + Bash |
| Dashboard | HTML/CSS/JS + Chart.js |
| CI/CD | GitHub Actions |
| Cloud | ghaymah.systems |
| Registry | Ghaymah Container Registry |

---

## License

This project was created for the Ghaymah Cloud Internship Program.