328 أسطر
9.6 KiB
Markdown
328 أسطر
9.6 KiB
Markdown
# Ghaymah Cloud Internship — SRE & SecOps Track
|
||
|
||
> Deployment, Monitoring, CI/CD, Scalability, and Observability on **ghaymah.systems**
|
||
|
||
**Author:** Mohamed Moustafa
|
||
**Track:** SRE & SecOps
|
||
**Platform:** [ghaymah.systems](https://ghaymah.systems)
|
||
|
||
---
|
||
|
||
## Project Overview
|
||
|
||
This repository contains all six tasks from the Ghaymah Cloud internship exam, covering the full lifecycle of cloud-native applications — from containerized deployment to post-incident analysis, CI/CD automation, scalability design, and real-time monitoring.
|
||
|
||
| Task | Topic | Score | Status |
|
||
|------|-------|-------|--------|
|
||
| Q1 | Deploy & Monitor on Ghaymah | 20/20 | Done |
|
||
| Q2 | Postmortem — OOMKilled Incident | 20/20 | Done |
|
||
| Q3 | CI/CD Pipeline with GitHub Actions | 20/20 | Done |
|
||
| Q4 | Scalability & Load Balancing | 20/20 | Done |
|
||
| Q5 | Mithal.space Monitoring Dashboard | 20/20 | Done |
|
||
| Q6 | Mortakaz Integration Proposals | 15/15 | Done |
|
||
|
||
---
|
||
|
||
## Repository Structure
|
||
|
||
```
|
||
ghaymah-exam-mohamed-sre/
|
||
├── README.md ← You are here
|
||
│
|
||
├── q1-deploy-monitor/
|
||
│ ├── Dockerfile ← Multi-stage Node.js build
|
||
│ ├── server.js ← Express API with /health endpoint
|
||
│ ├── package.json
|
||
│ ├── health-check.sh ← Bash monitor (30s interval)
|
||
│ ├── ghaymah.json ← Ghaymah CLI config
|
||
│ ├── architecture.svg ← Architecture diagram
|
||
│ └── public/
|
||
│ ├── dashboard.html ← Monitoring dashboard
|
||
│ ├── dashboard.css
|
||
│ └── dashboard.js
|
||
│
|
||
├── q2-postmortem/
|
||
│ ├── postmortem-report.md ← Full incident report
|
||
│ └── timeline.svg ← Visual incident timeline
|
||
│
|
||
├── q3-cicd/
|
||
│ ├── workflow.yml ← GitHub Actions workflow
|
||
│ ├── pipeline.svg ← CI/CD pipeline diagram
|
||
│ └── Readme.md ← Staging vs Production + CLI guide
|
||
│
|
||
├── q4-scalability/
|
||
│ ├── calculations.md ← Container math & strategy
|
||
│ ├── architecture.svg ← 15K req/s architecture
|
||
│ └── architecture.png ← PNG version
|
||
│
|
||
├── q5-mithal-monitor/
|
||
│ ├── monitor.py ← Python monitoring script
|
||
│ ├── dashboard.html ← Real-time dashboard
|
||
│ ├── style.css ← Dashboard styles
|
||
│ ├── script.js ← Dashboard logic
|
||
│ ├── Dockerfile ← Docker image (Python + Nginx)
|
||
│ ├── nginx.conf ← Nginx config
|
||
│ ├── entrypoint.sh ← Container entrypoint
|
||
│ ├── requirements.txt ← Python dependencies
|
||
│ ├── README.md ← Detailed documentation
|
||
│ └── screenshots/
|
||
│ ├── dashboard-preview.svg ← Dashboard preview
|
||
│ └── monitor-preview.svg ← Terminal preview
|
||
│
|
||
├── common-mortakaz/
|
||
│ ├── integration-1.md ← Circle Panel proposal
|
||
│ └── integration-2.md ← Bilya AI Assistant proposal
|
||
│
|
||
└── common-qabilah/
|
||
└── qabilah-profile.txt ← Qabilah profile link
|
||
```
|
||
|
||
---
|
||
|
||
## Q1 — Deploy & Monitor on Ghaymah
|
||
|
||
**Goal:** Containerize a Node.js API, deploy to ghaymah.systems, and monitor it.
|
||
|
||
### Architecture
|
||
|
||
|
||
|
||
### What Was Built
|
||
|
||
- **Node.js API** (Express) with `/health` endpoint returning status and request count
|
||
- **Multi-stage Dockerfile** — builder stage for `npm ci`, production stage on Alpine
|
||
- **Health check script** (`health-check.sh`) — polls `/health` every 30 seconds, logs status
|
||
- **Monitoring dashboard** — live status, response time, total requests
|
||
|
||
### Key Files
|
||
|
||
| File | Purpose |
|
||
|------|---------|
|
||
| `Dockerfile` | Multi-stage build: `node:22` → `node:22-alpine` |
|
||
| `server.js` | Express server with `/health` endpoint |
|
||
| `health-check.sh` | Bash monitoring loop (30s interval) |
|
||
| `ghaymah.json` | Ghaymah CLI configuration |
|
||
|
||
### Deploy Commands
|
||
|
||
```bash
|
||
# Install Ghaymah CLI
|
||
curl -sSL https://cli.ghaymah.systems/install.sh | bash
|
||
|
||
# Authenticate
|
||
$HOME/ghaymah/bin/gy auth login --email "EMAIL" --password "PW"
|
||
|
||
# Deploy
|
||
$HOME/ghaymah/bin/gy resource app launch
|
||
|
||
# Run monitor
|
||
chmod +x health-check.sh
|
||
./health-check.sh
|
||
```
|
||
|
||
---
|
||
|
||
## Q2 — Postmortem: OOMKilled Incident
|
||
|
||
**Goal:** Document a 45-minute outage caused by repeated OOMKilled events.
|
||
|
||
### Incident Timeline
|
||
|
||
|
||
|
||
### Summary
|
||
|
||
| Field | Detail |
|
||
|-------|--------|
|
||
| Duration | 45 minutes |
|
||
| Severity | High |
|
||
| Root Cause | Memory leak exceeded container limit |
|
||
| Impact | 100% request failure, 12+ restarts |
|
||
|
||
### Key Sections
|
||
|
||
- **Root Cause:** Application consumed more memory than the container's configured limit
|
||
- **Timeline:** Deploy → Memory spike → OOMKilled → Crash loop → Fix
|
||
- **Auto-Scaling Policy:** Scale-out at 80% memory, scale-in at 40%, min 2 / max 10 instances
|
||
- **Early Detection:** Prometheus + Grafana for memory metrics, alerts at 80% threshold
|
||
|
||
---
|
||
|
||
## Q3 — CI/CD Pipeline
|
||
|
||
**Goal:** Automate build, push, and deploy with manual approval gate.
|
||
|
||
### Pipeline
|
||
|
||
|
||
|
||
### Workflow Steps
|
||
|
||
1. **Trigger** — `git push` to `main` branch
|
||
2. **Build** — Docker image from Dockerfile
|
||
3. **Test** — `npm test` verification
|
||
4. **Push** — Image to Ghaymah Container Registry
|
||
5. **Approval** — Manual gate before production
|
||
6. **Deploy** — `gy resource app launch`
|
||
|
||
### Staging vs Production
|
||
|
||
| Feature | Staging | Production |
|
||
|---------|---------|------------|
|
||
| Users | Developers / QA | End Users |
|
||
| Data | Test Data | Production Data |
|
||
| Approval | Optional | Required |
|
||
| Stability | Medium | High |
|
||
|
||
### Ghaymah CLI Integration
|
||
|
||
```bash
|
||
# Install
|
||
curl -sSL https://cli.ghaymah.systems/install.sh | bash
|
||
|
||
# Login
|
||
gy auth login --email "EMAIL" --password "PW"
|
||
|
||
# Deploy
|
||
gy resource app launch
|
||
|
||
# Monitor
|
||
gy app logs
|
||
```
|
||
|
||
---
|
||
|
||
## Q4 — Scalability & Load Balancing
|
||
|
||
**Goal:** Design architecture for 15,000 req/s on Ghaymah Cloud.
|
||
|
||
### Architecture
|
||
|
||
|
||
|
||
### Calculations
|
||
|
||
```
|
||
Traffic: 15,000 req/s
|
||
Per Container: 500 req/s
|
||
Base Need: 15,000 / 500 = 30 containers
|
||
Safety Margin: 30% → 30 × 1.3 = 39 containers
|
||
```
|
||
|
||
### Cold Start Strategy
|
||
|
||
- Keep **2–3 warm containers** ready
|
||
- Use **lightweight Docker images** (Alpine-based)
|
||
- Configure **health checks** before routing traffic
|
||
- Trigger auto-scaling at **70% CPU** or **80% memory**
|
||
|
||
### Block Storage for Stateful Workloads
|
||
|
||
Ghaymah Block Storage persists data across container restarts:
|
||
|
||
- **Databases:** PostgreSQL, MySQL, MongoDB
|
||
- **File Storage:** User uploads, recordings
|
||
- **Logs:** Persistent application logs
|
||
- **Backups:** Automated backup storage
|
||
|
||
---
|
||
|
||
## Q5 — Mithal.space Monitoring Dashboard
|
||
|
||
**Goal:** Build a production-quality monitoring solution for mithal.space.
|
||
|
||
### Dashboard Preview
|
||
|
||
|
||
|
||
### Monitor Output
|
||
|
||
|
||
|
||
### Features
|
||
|
||
| Feature | Implementation |
|
||
|---------|---------------|
|
||
| HTTP Latency | `requests` library, measures GET response time |
|
||
| Uptime | Status code check (200–399 = UP) |
|
||
| SSL Certificate | `ssl` module, checks expiry date |
|
||
| DNS Lookup | `socket.getaddrinfo()` timing |
|
||
| Search Response | Real `/search?q=` request measurement |
|
||
| Data Storage | `metrics.json` (1,440 records ≈ 24h) |
|
||
| Dashboard | HTML/CSS/JS with Chart.js |
|
||
| Dark Mode | Toggle with localStorage persistence |
|
||
| CSV Export | One-click metrics export |
|
||
|
||
### Quick Start
|
||
|
||
```bash
|
||
cd q5-mithal-monitor
|
||
pip install -r requirements.txt
|
||
python monitor.py --interval 60
|
||
|
||
# Dashboard
|
||
python3 -m http.server 8080
|
||
# Open http://localhost:8080/dashboard.html
|
||
```
|
||
|
||
### Docker
|
||
|
||
```bash
|
||
docker build -t mithal-monitor .
|
||
docker run -d -p 8080:80 mithal-monitor
|
||
```
|
||
|
||
---
|
||
|
||
## Q6 — Mortakaz Integration Proposals
|
||
|
||
**Goal:** Propose integrations between mortakaz.com products and Ghaymah Cloud.
|
||
|
||
### Product 1: Circle Panel
|
||
|
||
User research platform for Arabic-speaking product teams.
|
||
|
||
- **Ghaymah Integration:** Containerized microservices (Frontend, Backend, AI Processing)
|
||
- **Block Storage:** Interview recordings and research data
|
||
- **mithal.space:** Searchable documentation and resources
|
||
|
||
### Product 2: Bilya AI Assistant
|
||
|
||
AI-powered virtual assistant for automotive service centers.
|
||
|
||
- **Ghaymah Integration:** Microservices (Frontend, API, AI, Booking)
|
||
- **Block Storage:** Customer conversations and booking records
|
||
- **mithal.space:** FAQ and documentation discoverability
|
||
|
||
### Recommendation
|
||
|
||
**Bilya AI Assistant** is the strongest candidate — its architecture naturally benefits from containers, auto-scaling, persistent storage, and CI/CD pipelines.
|
||
|
||
---
|
||
|
||
## Qabilah Profile
|
||
|
||
[qabilah.com/profile/mohamed-moustafa20](https://qabilah.com/profile/mohamed-moustafa20/)
|
||
|
||
---
|
||
|
||
## Technology Stack
|
||
|
||
| Layer | Technology |
|
||
|-------|-----------|
|
||
| Runtime | Node.js 22, Python 3.12 |
|
||
| Framework | Express.js |
|
||
| Container | Docker (multi-stage) |
|
||
| Web Server | Nginx |
|
||
| Monitoring | Python + Bash |
|
||
| Dashboard | HTML/CSS/JS + Chart.js |
|
||
| CI/CD | GitHub Actions |
|
||
| Cloud | ghaymah.systems |
|
||
| Registry | Ghaymah Container Registry |
|
||
|
||
---
|
||
|
||
## License
|
||
|
||
This project was created for the Ghaymah Cloud Internship Program.
|