commit 01b477b1394bb694cf05f6a0cda13e03596ad2ec Author: Anass_Ehab33 Date: Sun Jul 26 22:52:49 2026 +0300 feat: Ghayma Assessment - complete submission diff --git a/README.md b/README.md new file mode 100644 index 0000000..560b887 --- /dev/null +++ b/README.md @@ -0,0 +1,77 @@ +# Ghayma REST API Application + +A simple, lightweight Node.js Express REST API application with container support. + +## πŸš€ Features + +- **Express.js** lightweight web framework +- **CORS & JSON Middleware** pre-configured +- **Health Check Endpoint** (`/health`) for Docker/Kubernetes probes +- **Resource CRUD API** (`/api/v1/items`) +- **Docker Support** with health checks and optimized `.dockerignore` + +--- + +## πŸ› οΈ Endpoints Overview + +| Method | Endpoint | Description | +| :--- | :--- | :--- | +| `GET` | `/` | API Metadata & Welcome Info | +| `GET` | `/health` | Application Health & Uptime Status | +| `GET` | `/api/v1/items` | List all items | +| `GET` | `/api/v1/items/:id` | Get item by ID | +| `POST` | `/api/v1/items` | Create new item | +| `DELETE` | `/api/v1/items/:id` | Delete item by ID | + +--- + +## πŸ’» Running Locally + +### Prerequisites +- Node.js (v18+) + +### Steps +1. Install dependencies: + ```bash + npm install + ``` + +2. Start the API server: + ```bash + npm start + ``` + Or for development auto-reload (Node 18+): + ```bash + npm run dev + ``` + +3. Test the server: + Navigate to `http://localhost:3000` or run: + ```bash + curl http://localhost:3000/health + ``` + +--- + +## 🐳 Running with Docker + +### Build the Docker Image +```bash +docker build -t ghayma-api . +``` + +### Run the Container +```bash +docker run -d -p 3000:3000 --name ghayma-api-app ghayma-api +``` + +### Test Containerized API +```bash +curl http://localhost:3000/api/v1/items +``` + +### Stop Container +```bash +docker stop ghayma-api-app +docker rm ghayma-api-app +``` diff --git a/common-mortakaz/product-integration-report.md b/common-mortakaz/product-integration-report.md new file mode 100644 index 0000000..380083d --- /dev/null +++ b/common-mortakaz/product-integration-report.md @@ -0,0 +1,289 @@ +# Mortakaz Product Discovery & Cloud Integration Proposals + +**Source Platform:** [mortakaz.com](https://mortakaz.com) (Ω…Ψ±ΨͺΩƒΨ² β€” Arabic Products Platform) +**Assessment:** Ghayma Cloud Integration Analysis +**Date:** July 26, 2026 + +--- + +## Products Explored on Mortakaz + +After browsing mortakaz.com, the following products were examined: + +| # | Product | Category | Cloud Potential | +| :--- | :--- | :--- | :--- | +| 1 | **Taylor CV** (ΨͺΨ§ΩŠΩ„ΩˆΨ± سي في) | Employment & AI | ⭐⭐⭐⭐⭐ | +| 2 | **Athar** (Ψ£Ψ«Ψ±) | Islamic App | ⭐⭐ | +| 3 | **Code City** (ΩƒΩˆΨ― سيΨͺي) | Developer Tools / 3D Visualization | ⭐⭐⭐⭐ | +| 4 | **Social Browser** (Ψ³ΩˆΨ΄ΩŠΨ§Ω„ براوزر) | Web Browsers | ⭐⭐ | +| 5 | **Workflow** (ΩˆΩˆΨ±Ωƒ ΩΩ„Ωˆ) | Enterprise SaaS | ⭐⭐⭐⭐⭐ | + +### Selected Products for Integration Proposals + +**Product A:** πŸ† **Workflow (ΩˆΩˆΨ±Ωƒ ΩΩ„Ωˆ)** β€” Enterprise SaaS Platform +**Product B:** πŸ₯ˆ **Taylor CV (ΨͺΨ§ΩŠΩ„ΩˆΨ± سي في)** β€” AI-Powered Career Assistant + +These two were selected because they are **cloud-native SaaS applications** with high computational demands, data persistence needs, and clear scalability requirements β€” making them ideal candidates for cloud platform integration. + +--- + +--- + +# Integration Proposal 1: Workflow (ΩˆΩˆΨ±Ωƒ ΩΩ„Ωˆ) + +## Product Description + +**Workflow** is a centralized, multi-tenant SaaS platform designed to unify company management, project tracking, and financial operations into a single dashboard. It targets small-to-medium enterprises (SMEs) and holding companies with multiple subsidiaries. + +**Key Features:** +- Task & project management with Kanban boards +- Invoicing, payroll, and financial reporting +- Multi-tenant architecture (each company is isolated) +- Employee management and HR operations +- Real-time collaboration across teams and departments + +**Source:** [mortakaz.com/projects/6a2eed27ec1d16fce885476a](https://www.mortakaz.com/projects/6a2eed27ec1d16fce885476a) + +--- + +## Integration with Ghayma Cloud (ghaymah.systems) + +### How to Integrate + +Workflow can be deployed and scaled on Ghayma Cloud's container platform with persistent Block Storage for its multi-tenant data layer. The integration leverages three core Ghayma services: + +1. **Ghayma Container Platform** β€” Host Workflow's backend API and frontend as containerized microservices +2. **Ghayma Block Storage** β€” Persistent SSD volumes for PostgreSQL database (tenant data, invoices, payroll) +3. **Ghayma Container Registry** β€” Store and version Docker images via CI/CD pipeline + +### Architecture Sketch + +``` +β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” +β”‚ GHAYMA CLOUD PLATFORM β”‚ +β”‚ β”‚ +β”‚ β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”‚ +β”‚ β”‚ NETWORKING LAYER β”‚ β”‚ +β”‚ β”‚ β”‚ β”‚ +β”‚ β”‚ Users ──▢ [DNS] ──▢ [CDN] ──▢ [Load Balancer (L7)] β”‚ β”‚ +β”‚ β”‚ β”‚ β”‚ β”‚ +β”‚ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β”‚ +β”‚ β”‚ β”‚ +β”‚ β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”‚ +β”‚ β”‚ COMPUTE LAYER β”‚ β”‚ +β”‚ β”‚ β”‚ β”‚ +β”‚ β”‚ β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”‚ β”‚ +β”‚ β”‚ β”‚ Frontend β”‚ β”‚ Backend API β”‚ β”‚ Worker Service β”‚ β”‚ β”‚ +β”‚ β”‚ β”‚ (Vue/Nuxt) β”‚ β”‚ (Node.js / β”‚ β”‚ (Background β”‚ β”‚ β”‚ +β”‚ β”‚ β”‚ 3 replicas β”‚ β”‚ Python) β”‚ β”‚ jobs: payroll, β”‚ β”‚ β”‚ +β”‚ β”‚ β”‚ β”‚ β”‚ 5 replicas β”‚ β”‚ reports, email)β”‚ β”‚ β”‚ +β”‚ β”‚ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β””β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”˜ β””β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β”‚ β”‚ +β”‚ β”‚ β”‚ β”‚ β”‚ β”‚ +β”‚ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β”‚ +β”‚ β”‚ β”‚ β”‚ +β”‚ β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”‚ +β”‚ β”‚ DATA LAYER β”‚ β”‚ +β”‚ β”‚ β”‚ β”‚ +β”‚ β”‚ β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”‚ β”‚ +β”‚ β”‚ β”‚ PostgreSQL DB β”‚ β”‚ Redis β”‚ β”‚ Ghayma β”‚ β”‚ β”‚ +β”‚ β”‚ β”‚ (Multi-tenant) β”‚ β”‚ (Cache + β”‚ β”‚ Block β”‚ β”‚ β”‚ +β”‚ β”‚ β”‚ β”‚ β”‚ Sessions) β”‚ β”‚ Storage β”‚ β”‚ β”‚ +β”‚ β”‚ β”‚ β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”‚ β”‚ β”‚ β”‚ β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”‚ β”‚ β”‚ +β”‚ β”‚ β”‚ β”‚ Tenant A Data β”‚ β”‚ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β”‚ β”‚ DB Vol β”‚ β”‚ β”‚ β”‚ +β”‚ β”‚ β”‚ β”‚ Tenant B Data β”‚ β”‚ β”‚ β”‚ 200GB β”‚ β”‚ β”‚ β”‚ +β”‚ β”‚ β”‚ β”‚ Tenant C Data β”‚ β”‚ β”‚ β”‚ SSD β”‚ β”‚ β”‚ β”‚ +β”‚ β”‚ β”‚ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β”‚ β”‚ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β”‚ β”‚ β”‚ +β”‚ β”‚ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β”‚ β”‚ +β”‚ β”‚ β”‚ β”‚ β”‚ +β”‚ β”‚ β–Ό β”‚ β”‚ +β”‚ β”‚ Ghayma Block Storage Volume (persistent across restarts) β”‚ β”‚ +β”‚ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β”‚ +β”‚ β”‚ +β”‚ β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”‚ +β”‚ β”‚ CI/CD: GitHub Actions β†’ Ghayma Registry β”‚ β”‚ +β”‚ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β”‚ +β”‚ β”‚ +β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ +``` + +--- + +## Value Added to End Users + +| Value | Impact | +| :--- | :--- | +| **High Availability** | Multi-replica deployment ensures 99.9% uptime for all tenants | +| **Data Sovereignty** | Hosting on Ghayma (regional Middle East cloud) keeps Arabic enterprise data within the region, complying with local regulations | +| **Auto-Scaling** | Handle tenant growth automatically β€” from 10 to 10,000 companies without downtime | +| **Persistent Data** | Block Storage guarantees financial records (invoices, payroll) survive any container crash or deployment | +| **Faster Performance** | Regional cloud = lower latency for MENA-based SMEs vs. US/EU-hosted alternatives | +| **Cost Efficiency** | Pay-per-use container pricing scales with actual tenant load instead of fixed server costs | + +--- + +## Potential Challenges + +| Challenge | Category | Mitigation | +| :--- | :--- | :--- | +| **Multi-tenant data isolation** | Technical | Use PostgreSQL schemas-per-tenant or row-level security policies | +| **Database backup & DR** | Technical | Schedule Ghayma Block Storage snapshots every 6 hours | +| **Compliance (financial data)** | Commercial | Ensure Ghayma Cloud meets local data residency regulations for financial records | +| **Vendor lock-in** | Commercial | Keep application container-based (Docker) for portability to other cloud providers | +| **Cold start latency** | Technical | Use pre-warmed container pool (min 2 replicas always running) | +| **Integration complexity** | Technical | Gradual migration: start with backend API on Ghayma, keep DB local initially | + +--- + +--- + +# Integration Proposal 2: Taylor CV (ΨͺΨ§ΩŠΩ„ΩˆΨ± سي في) + +## Product Description + +**Taylor CV** is an AI-powered career assistant that helps Arabic-speaking job seekers land their target roles. It uses the user's knowledge base (skills, work history, projects) combined with a specific job description to generate optimized, ATS-friendly resumes and cover letters. + +**Key Features:** +- AI-generated customized resumes and cover letters +- Interview preparation (Q&A + mock voice calls) +- Daily learning plans to bridge skill gaps +- Application tracking board +- Multi-language support: Arabic, English, French +- Bring-your-own API key option for free access + +**Source:** [mortakaz.com/projects/6a60ad7e3d576288c1011ff2](https://www.mortakaz.com/projects/6a60ad7e3d576288c1011ff2) + +--- + +## Integration with Ghayma Cloud & mithal.space + +### How to Integrate + +Taylor CV can be deployed on Ghayma Cloud with its AI processing pipeline leveraging containerized microservices. The integration with **mithal.space** (Ghayma's example/demo engine) can provide a public showcase and monitoring dashboard for the service. + +1. **Ghayma Container Platform** β€” Host the web app, API, and AI worker services +2. **Ghayma Block Storage** β€” Store user profiles, generated documents (PDFs), and conversation history +3. **mithal.space Integration** β€” Publish Taylor CV as a featured example on mithal.space with live uptime monitoring, demonstrating Ghayma's AI workload capabilities + +### Architecture Sketch + +``` +β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” +β”‚ GHAYMA CLOUD PLATFORM β”‚ +β”‚ β”‚ +β”‚ Users (Job Seekers) β”‚ +β”‚ β”‚ β”‚ +β”‚ β–Ό β”‚ +β”‚ β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”‚ +β”‚ β”‚ FRONTEND LAYER β”‚ β”‚ +β”‚ β”‚ Taylor CV Web App (Vue/React) β”‚ β”‚ +β”‚ β”‚ Hosted on Ghayma CDN + Container β”‚ β”‚ +β”‚ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β”‚ +β”‚ β”‚ β”‚ +β”‚ β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”‚ +β”‚ β”‚ API GATEWAY β”‚ β”‚ +β”‚ β”‚ β”‚ β”‚ β”‚ +β”‚ β”‚ β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”‚ β”‚ +β”‚ β”‚ β”‚ β”‚ β”‚ β”‚ β”‚ +β”‚ β”‚ β–Ό β–Ό β–Ό β”‚ β”‚ +β”‚ β”‚ β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”‚ β”‚ +β”‚ β”‚ β”‚ Auth β”‚ β”‚ Resume β”‚ β”‚ Interview β”‚ β”‚ β”‚ +β”‚ β”‚ β”‚ Service β”‚ β”‚ Builder β”‚ β”‚ Prep Serviceβ”‚ β”‚ β”‚ +β”‚ β”‚ β”‚ β”‚ β”‚ Service β”‚ β”‚ (Voice AI) β”‚ β”‚ β”‚ +β”‚ β”‚ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β””β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”˜ β””β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”˜ β”‚ β”‚ +β”‚ β”‚ β”‚ β”‚ β”‚ β”‚ +β”‚ β”‚ β–Ό β–Ό β”‚ β”‚ +β”‚ β”‚ β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”‚ β”‚ +β”‚ β”‚ β”‚ AI Processing Queue β”‚ β”‚ β”‚ +β”‚ β”‚ β”‚ (Background Workers) β”‚ β”‚ β”‚ +β”‚ β”‚ β”‚ β€’ PDF Generation β”‚ β”‚ β”‚ +β”‚ β”‚ β”‚ β€’ LLM API Calls β”‚ β”‚ β”‚ +β”‚ β”‚ β”‚ β€’ Voice Transcription β”‚ β”‚ β”‚ +β”‚ β”‚ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β”‚ β”‚ +β”‚ β”‚ β”‚ β”‚ β”‚ +β”‚ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β”‚ +β”‚ β”‚ β”‚ +β”‚ β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”‚ +β”‚ β”‚ DATA LAYER β”‚ β”‚ +β”‚ β”‚ β”‚ β”‚ +β”‚ β”‚ β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”‚ β”‚ +β”‚ β”‚ β”‚ MongoDB / β”‚ β”‚ Redis β”‚ β”‚ Ghayma Block Storage β”‚ β”‚ β”‚ +β”‚ β”‚ β”‚ PostgreSQL β”‚ β”‚ (Cache β”‚ β”‚ β€’ User profiles β”‚ β”‚ β”‚ +β”‚ β”‚ β”‚ (User data, β”‚ β”‚ + Job β”‚ β”‚ β€’ Generated PDFs β”‚ β”‚ β”‚ +β”‚ β”‚ β”‚ resumes) β”‚ β”‚ Queue) β”‚ β”‚ β€’ Voice recordings β”‚ β”‚ β”‚ +β”‚ β”‚ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β”‚ β”‚ +β”‚ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β”‚ +β”‚ β”‚ +β”‚ β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”‚ +β”‚ β”‚ mithal.space SHOWCASE INTEGRATION β”‚ β”‚ +β”‚ β”‚ β”‚ β”‚ +β”‚ β”‚ β€’ Live demo page on mithal.space featuring Taylor CV β”‚ β”‚ +β”‚ β”‚ β€’ Uptime monitoring dashboard (99.9% SLA display) β”‚ β”‚ +β”‚ β”‚ β€’ Public API status page for job seekers β”‚ β”‚ +β”‚ β”‚ β€’ Case study: "AI Workloads on Ghayma Cloud" β”‚ β”‚ +β”‚ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β”‚ +β”‚ β”‚ +β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ +``` + +--- + +## Value Added to End Users + +| Value | Impact | +| :--- | :--- | +| **Low Latency AI** | Running AI workers on Ghayma's regional cloud reduces round-trip time for Arabic NLP processing | +| **Scalable AI Workers** | Auto-scale GPU/CPU workers during peak job application seasons (graduation, Ramadan hiring) | +| **Data Privacy** | Users' personal career data (skills, work history) stays in-region, not on US servers | +| **Reliable PDF Generation** | Block Storage ensures generated resumes are never lost, even during worker crashes | +| **mithal.space Showcase** | Public visibility as a featured Ghayma Cloud success story drives user adoption | +| **Cost-Effective AI** | Container-based workers spin up only when processing requests, reducing idle compute costs | + +--- + +## Potential Challenges + +| Challenge | Category | Mitigation | +| :--- | :--- | :--- | +| **LLM API costs** | Commercial | Implement usage tiers and the "bring your own API key" model to offload costs | +| **Arabic NLP accuracy** | Technical | Fine-tune models on Arabic resume datasets; leverage community feedback from mortakaz | +| **GPU availability** | Technical | Pre-reserve GPU instances on Ghayma Cloud; fall back to CPU for smaller workloads | +| **Voice AI latency** | Technical | Deploy WebSocket-based streaming for real-time mock interviews | +| **User data sensitivity** | Commercial | Implement end-to-end encryption for stored resumes and personal data | +| **Peak load spikes** | Technical | Pre-scale workers during known peak periods (September, January hiring seasons) | + +--- + +--- + +# Final Recommendation: Most Viable Product + +## πŸ† Winner: Workflow (ΩˆΩˆΨ±Ωƒ ΩΩ„Ωˆ) + +### Why Workflow is the Most Viable Integration + +| Criteria | Workflow | Taylor CV | +| :--- | :--- | :--- | +| **Revenue Model** | βœ… SaaS subscriptions (recurring, predictable) | ⚠️ Freemium with API-key model (uncertain revenue) | +| **Cloud Dependency** | βœ… Critical β€” needs persistent DB, multi-tenancy, scaling | ⚠️ Moderate β€” AI can run on any cloud | +| **Data Persistence Need** | βœ… Must-have β€” financial records, invoices, payroll | ⚠️ Nice-to-have β€” resumes can be re-generated | +| **Target Market Size** | βœ… Large β€” every SME in MENA region (millions) | ⚠️ Niche β€” job seekers (cyclical demand) | +| **Regulatory Fit** | βœ… Financial data residency laws favor regional cloud | ⚠️ Less regulatory pressure | +| **Long-term Stickiness** | βœ… High β€” companies rarely migrate ERP/management tools | ⚠️ Low β€” users may switch AI tools easily | +| **Ghayma Revenue Potential** | βœ… High β€” persistent storage + compute + DB | ⚠️ Moderate β€” burst compute only | + +### Summary + +**Workflow** is the most viable because: + +1. **It needs the cloud** β€” A multi-tenant SaaS managing invoices, payroll, and employee data **requires** reliable cloud infrastructure with persistent storage. This isn't optional; it's fundamental. + +2. **Recurring revenue for both parties** β€” Workflow charges monthly SaaS fees, and Ghayma earns from persistent compute + storage. Both revenue streams are predictable. + +3. **Data residency is a competitive advantage** β€” MENA enterprises increasingly require financial data to stay in-region. Hosting on Ghayma Cloud (regional) is a selling point competitors on AWS US-East can't match. + +4. **High stickiness** β€” Once a company moves its invoicing, HR, and project management to Workflow on Ghayma, switching costs are extremely high, creating a long-term customer for both Workflow and Ghayma Cloud. + +--- + +> **Report prepared for:** Ghayma Assessment β€” Product Discovery & Integration section +> **Last updated:** July 26, 2026 diff --git a/common-qabilah/qabilah-profile.txt b/common-qabilah/qabilah-profile.txt new file mode 100644 index 0000000..e69de29 diff --git a/q1-deploy-monitor/Dockerfile b/q1-deploy-monitor/Dockerfile new file mode 100644 index 0000000..ad36c50 --- /dev/null +++ b/q1-deploy-monitor/Dockerfile @@ -0,0 +1,12 @@ +FROM node:20-alpine + +WORKDIR /app + +COPY package*.json ./ +RUN npm ci --only=production + +COPY . . + +EXPOSE 3000 + +CMD ["npm","start"] diff --git a/q1-deploy-monitor/Health-check.py b/q1-deploy-monitor/Health-check.py new file mode 100644 index 0000000..2bc0288 --- /dev/null +++ b/q1-deploy-monitor/Health-check.py @@ -0,0 +1,148 @@ +#!/usr/bin/env python3 +""" +Monitoring Script for Ghayma REST API +Scans the application health endpoint every 30 seconds and logs results. +""" + +import urllib.request +import urllib.error +import json +import time +import datetime +import os +import sys + +# Configuration +APP_URL = os.environ.get("APP_URL", "http://localhost:3000") +HEALTH_ENDPOINT = f"{APP_URL}/health" +CHECK_INTERVAL = 30 # seconds +LOG_FILE = os.path.join(os.path.dirname(os.path.abspath(__file__)), "monitor.log") +TIMEOUT = 5 # seconds + +# ANSI colors for terminal output +GREEN = "\033[92m" +RED = "\033[91m" +YELLOW = "\033[93m" +CYAN = "\033[96m" +RESET = "\033[0m" +BOLD = "\033[1m" + + +def log(message, level="INFO"): + """Log a message to both console and log file.""" + timestamp = datetime.datetime.now().strftime("%Y-%m-%d %H:%M:%S") + log_entry = f"[{timestamp}] [{level}] {message}" + + # Color-coded console output + if level == "OK": + print(f"{GREEN}{log_entry}{RESET}") + elif level == "ERROR": + print(f"{RED}{BOLD}{log_entry}{RESET}") + elif level == "WARN": + print(f"{YELLOW}{log_entry}{RESET}") + else: + print(f"{CYAN}{log_entry}{RESET}") + + # Append to log file + with open(LOG_FILE, "a", encoding="utf-8") as f: + f.write(log_entry + "\n") + + +def check_health(): + """Send a GET request to the health endpoint and return the result.""" + try: + req = urllib.request.Request(HEALTH_ENDPOINT, method="GET") + start_time = time.time() + response = urllib.request.urlopen(req, timeout=TIMEOUT) + response_time_ms = round((time.time() - start_time) * 1000) + status_code = response.status + body = json.loads(response.read().decode("utf-8")) + + return { + "healthy": status_code == 200 and body.get("status") == "UP", + "status_code": status_code, + "response_time_ms": response_time_ms, + "uptime": body.get("uptime"), + "error": None, + } + + except urllib.error.HTTPError as e: + return { + "healthy": False, + "status_code": e.code, + "response_time_ms": None, + "uptime": None, + "error": f"HTTP {e.code}: {e.reason}", + } + + except urllib.error.URLError as e: + return { + "healthy": False, + "status_code": None, + "response_time_ms": None, + "uptime": None, + "error": f"Connection failed: {e.reason}", + } + + except Exception as e: + return { + "healthy": False, + "status_code": None, + "response_time_ms": None, + "uptime": None, + "error": str(e), + } + + +def main(): + consecutive_failures = 0 + + print(f"{BOLD}{CYAN}") + print("=" * 55) + print(" Ghayma REST API - Health Monitor") + print(f" Target: {HEALTH_ENDPOINT}") + print(f" Interval: {CHECK_INTERVAL}s") + print(f" Log file: {LOG_FILE}") + print("=" * 55) + print(f"{RESET}") + + log(f"Monitor started | Target: {HEALTH_ENDPOINT} | Interval: {CHECK_INTERVAL}s") + + try: + while True: + result = check_health() + + if result["healthy"]: + consecutive_failures = 0 + uptime_str = f"{result['uptime']:.1f}s" if result["uptime"] else "N/A" + log( + f"HEALTHY | Status: {result['status_code']} | " + f"Response: {result['response_time_ms']}ms | " + f"Uptime: {uptime_str}", + level="OK", + ) + else: + consecutive_failures += 1 + log( + f"UNHEALTHY | Error: {result['error']} | " + f"Consecutive failures: {consecutive_failures}", + level="ERROR", + ) + + if consecutive_failures >= 3: + log( + f"ALERT: {consecutive_failures} consecutive failures! " + f"Application may be DOWN.", + level="WARN", + ) + + time.sleep(CHECK_INTERVAL) + + except KeyboardInterrupt: + log("Monitor stopped by user.", level="INFO") + print(f"\n{YELLOW}Monitor stopped.{RESET}") + sys.exit(0) + + +if __name__ == "__main__": + main() diff --git a/q1-deploy-monitor/dashboard.html b/q1-deploy-monitor/dashboard.html new file mode 100644 index 0000000..f934256 --- /dev/null +++ b/q1-deploy-monitor/dashboard.html @@ -0,0 +1,764 @@ + + + + + + Ghayma API β€” Monitoring Dashboard + + + + + +
+ +
+
+ +

Ghayma Monitoring

+
+
+
+
+ CONNECTING... +
+
Updated: β€”
+
+
+ + +
+ +
+
+
πŸ“Š
+
Total Requests
+
0
+
0 req/min
+
+
+
βœ…
+
Successful
+
0
+
0% success rate
+
+
+
⚑
+
Avg Response Time
+
0 ms
+
Last 100 requests
+
+
+
❌
+
Errors
+
0
+
0% error rate
+
+
+ + +
+
+
+
⏱️ Response Time (last 20 requests)
+
+
+
+
Waiting for data...
+
+
+
+ +
+
+
πŸ”— Requests by Endpoint
+
+
+
    +
  • No requests yet
  • +
+
+
+
+ + +
+
+
+
πŸ• Uptime
+
+
+
+
0
Days
+
0
Hours
+
0
Minutes
+
0
Seconds
+
+
+
+ +
+
+
πŸ“‹ Status Codes
+
+
+
+
+
0
+
2xx Success
+
+
+
0
+
3xx Redirect
+
+
+
0
+
4xx Client Err
+
+
+
0
+
5xx Server Err
+
+
+
+
+
+
+
+ + + + diff --git a/q2-postmortem/postmortem-report.md b/q2-postmortem/postmortem-report.md new file mode 100644 index 0000000..84eb85f --- /dev/null +++ b/q2-postmortem/postmortem-report.md @@ -0,0 +1,387 @@ +# πŸ”΄ Incident Postmortem Report β€” OOMKilled Application Crash + +**Incident ID:** INC-2026-0726 +**Severity:** P1 β€” Critical +**Duration:** 45 minutes +**Date:** July 26, 2026 +**Author:** DevOps Team +**Status:** Resolved + +--- + +## 1. Executive Summary + +On July 26, 2026, the **Ghayma REST API** application hosted on a cloud container platform experienced a **45-minute outage** caused by repeated **OOMKilled** (Out of Memory Killed) events. The container runtime terminated the application process multiple times after it exceeded its allocated memory limit. Each restart triggered the same memory spike, creating a crash loop that prevented the service from recovering. + +**Impact:** +- 100% service unavailability for 45 minutes +- All API consumers (frontend clients, monitoring, health probes) received connection errors +- Approximately **2,700 failed requests** during the outage window (estimated ~60 req/min baseline) +- Container orchestrator marked the pod/container as `CrashLoopBackOff` after repeated restart failures + +--- + +## 2. Incident Timeline + +| Time (UTC) | Event | +| :--- | :--- | +| **11:00** | 🟒 Application running normally. Memory usage stable at ~120MB (limit: 256MB) | +| **11:12** | πŸ“ˆ Traffic spike begins β€” external batch job sends large payload requests to `POST /api/v1/items` | +| **11:15** | ⚠️ Memory usage crosses **200MB** (78% of limit). No alerts triggered | +| **11:18** | πŸ”΄ Memory hits **256MB** limit. Kernel OOM killer terminates the container process (`OOMKilled`) | +| **11:18** | πŸ”„ Container runtime automatically restarts the container (Restart #1) | +| **11:20** | πŸ”΄ Application starts, loads cached data into memory, immediately OOMKilled again (Restart #2) | +| **11:20–11:45** | πŸ” **Crash loop** β€” container restarts 12 times. Orchestrator applies exponential backoff (`CrashLoopBackOff`) | +| **11:32** | πŸ“Ÿ On-call engineer alerted via PagerDuty after health check failures exceed 10 minutes | +| **11:38** | πŸ” Engineer identifies OOMKilled events in container logs and platform event stream | +| **11:45** | πŸ› οΈ Engineer increases memory limit from **256MB β†’ 512MB** and deploys hotfix | +| **11:48** | πŸ”„ Application restarts successfully. Memory stabilizes at ~180MB | +| **11:50** | πŸ“‰ Batch job completes. Traffic returns to normal levels | +| **12:03** | 🟒 Service fully confirmed stable. Incident resolved | + +**Total downtime:** 45 minutes (11:18 β€” 12:03 UTC) + +--- + +## 3. Root Cause Analysis + +### 3.1 Direct Cause + +The container was configured with a **memory limit of 256MB**, which was insufficient to handle traffic spikes. When an external batch job sent a burst of large `POST` requests with sizable JSON payloads, the Node.js process accumulated in-memory data (parsed request bodies, in-memory item storage, response buffers) beyond the container's limit. + +The Linux kernel's **OOM Killer** terminated the process when it attempted to allocate memory beyond the `256MB` cgroup limit. + +### 3.2 Contributing Factors + +``` +β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” +β”‚ ROOT CAUSE BREAKDOWN β”‚ +β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€ +β”‚ β”‚ +β”‚ 1. INSUFFICIENT MEMORY LIMIT β”‚ +β”‚ └─ Container limited to 256MB (too tight for Node.js) β”‚ +β”‚ β”‚ +β”‚ 2. NO MEMORY-AWARE AUTO-SCALING β”‚ +β”‚ └─ Only 1 replica running, no HPA/scaling policy β”‚ +β”‚ β”‚ +β”‚ 3. UNBOUNDED IN-MEMORY DATA STORE β”‚ +β”‚ └─ Items array grows indefinitely with no cap β”‚ +β”‚ β”‚ +β”‚ 4. NO MEMORY ALERTS / EARLY WARNING β”‚ +β”‚ └─ Alert only triggered after 10 min of health failures β”‚ +β”‚ β”‚ +β”‚ 5. NO REQUEST PAYLOAD SIZE LIMIT β”‚ +β”‚ └─ Express accepted arbitrarily large JSON bodies β”‚ +β”‚ β”‚ +β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ +``` + +### 3.3 Why Did Restarts Fail to Recover? + +Each restart reloaded the same conditions: +1. Container starts β†’ Node.js initializes (~80MB baseline) +2. Pending queued requests immediately hit the API +3. Large payloads + in-memory storage push memory past the limit +4. OOMKilled again within seconds +5. Orchestrator enters **CrashLoopBackOff** with increasing delays between restarts + +--- + +## 4. Remediation Actions Taken + +| Action | Status | +| :--- | :--- | +| Increased container memory limit to 512MB | βœ… Done (hotfix) | +| Restarted container with new limits | βœ… Done | +| Confirmed service recovery and stability | βœ… Done | + +--- + +## 5. Recommendations & Prevention + +### 5.1 Application-Level Fixes + +#### A. Limit Request Payload Size +```javascript +// In src/index.js β€” restrict incoming JSON body size +app.use(express.json({ limit: '1mb' })); +``` +> Prevents a single large request from consuming excessive memory. + +#### B. Cap In-Memory Data Store +```javascript +// Limit the items array to prevent unbounded growth +const MAX_ITEMS = 1000; + +app.post('/api/v1/items', (req, res) => { + if (items.length >= MAX_ITEMS) { + return res.status(429).json({ + success: false, + error: 'Maximum item limit reached' + }); + } + // ... rest of handler +}); +``` + +#### C. Set Node.js Memory Ceiling +```dockerfile +# In Dockerfile β€” set explicit V8 heap limit +CMD ["node", "--max-old-space-size=384", "src/index.js"] +``` +> Ensures Node.js garbage collector runs more aggressively before hitting the container limit. + +#### D. Use External Storage for Production +Replace the in-memory `items` array with a proper database (Redis, PostgreSQL, MongoDB) so application memory remains constant regardless of data volume. + +--- + +### 5.2 Container & Infrastructure Fixes + +#### A. Set Proper Resource Requests & Limits + +```yaml +# Kubernetes Deployment example +resources: + requests: + memory: "256Mi" # Guaranteed minimum + cpu: "100m" + limits: + memory: "512Mi" # Hard ceiling + cpu: "500m" +``` + +> [!IMPORTANT] +> **Rule of thumb:** Set `limits.memory` to at least **2x** the average working set. Set `requests.memory` to the **steady-state average**. + +#### B. Add Graceful Shutdown Handling +```javascript +process.on('SIGTERM', () => { + console.log('SIGTERM received. Shutting down gracefully...'); + server.close(() => { + process.exit(0); + }); +}); +``` + +--- + +## 6. Auto-Scaling Policy Design + +### 6.1 Horizontal Pod Autoscaler (HPA) Policy + +The following auto-scaling policy prevents a single container from being overwhelmed by distributing load across multiple replicas: + +```yaml +apiVersion: autoscaling/v2 +kind: HorizontalPodAutoscaler +metadata: + name: ghayma-api-hpa + namespace: production +spec: + scaleTargetRef: + apiVersion: apps/v1 + kind: Deployment + name: ghayma-api + + # Replica bounds + minReplicas: 2 # Always run at least 2 for high availability + maxReplicas: 10 # Cap to control costs + + # Scaling metrics + metrics: + # Scale based on MEMORY usage (primary - prevents OOMKill) + - type: Resource + resource: + name: memory + target: + type: Utilization + averageUtilization: 70 # Scale up when memory > 70% of limit + + # Scale based on CPU usage (secondary) + - type: Resource + resource: + name: cpu + target: + type: Utilization + averageUtilization: 75 # Scale up when CPU > 75% + + # Scaling behavior (prevents flapping) + behavior: + scaleUp: + stabilizationWindowSeconds: 30 # React quickly to spikes + policies: + - type: Pods + value: 2 # Add up to 2 pods at a time + periodSeconds: 60 + scaleDown: + stabilizationWindowSeconds: 300 # Wait 5 min before scaling down + policies: + - type: Pods + value: 1 # Remove 1 pod at a time + periodSeconds: 120 +``` + +### 6.2 How This Policy Prevents Duplication of the Incident + +``` +Traffic Spike Detected + β”‚ + β–Ό +β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” +β”‚ Memory usage > 70% │──── YES ──▢ HPA adds new replicas (up to 10) +β”‚ on existing pods? β”‚ Load balancer distributes traffic +β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ Memory per-pod stays under limit + β”‚ NO β”‚ + β–Ό β–Ό + Normal operation Spike absorbed across multiple pods + No single pod reaches OOMKill threshold +``` + +### 6.3 Cloud Platform Auto-Scaling (Non-Kubernetes) + +For managed container platforms (e.g., Ghayma Cloud, AWS ECS, Google Cloud Run): + +| Setting | Value | Rationale | +| :--- | :--- | :--- | +| **Min instances** | 2 | Avoid cold starts & single point of failure | +| **Max instances** | 10 | Cost ceiling | +| **Scale-up trigger** | Memory > 70% OR CPU > 75% OR Concurrent requests > 50 | Multi-signal scaling | +| **Scale-up cooldown** | 30 seconds | React quickly to spikes | +| **Scale-down cooldown** | 5 minutes | Prevent flapping during variable traffic | +| **Memory per instance** | 512MB | 2x steady-state working set | + +--- + +## 7. Early Detection β€” Monitoring & Alerting Strategy + +### 7.1 Key Metrics to Monitor + +| Metric | Source | Warning Threshold | Critical Threshold | +| :--- | :--- | :--- | :--- | +| **Container Memory Usage** | cAdvisor / Platform metrics | > 70% of limit | > 85% of limit | +| **Container Restart Count** | Kubelet / Platform events | β‰₯ 1 restart in 5 min | β‰₯ 3 restarts in 10 min | +| **OOMKilled Events** | Kernel / Container runtime | Any occurrence | N/A (always critical) | +| **Response Time (p95)** | Application `/metrics` endpoint | > 500ms | > 2000ms | +| **Error Rate (5xx)** | Load balancer / Application | > 1% | > 5% | +| **Health Check Failures** | Platform health probe | 1 consecutive failure | 3 consecutive failures | +| **Request Queue Depth** | Load balancer | > 50 pending | > 200 pending | + +### 7.2 Alerting Rules (Prometheus / Grafana Example) + +```yaml +# Alert: Memory approaching container limit +- alert: HighMemoryUsage + expr: | + (container_memory_usage_bytes{container="ghayma-api"} + / container_spec_memory_limit_bytes{container="ghayma-api"}) > 0.70 + for: 2m + labels: + severity: warning + annotations: + summary: "Ghayma API memory usage above 70%" + description: "Container {{ $labels.pod }} memory at {{ $value | humanizePercentage }}" + +# Alert: OOMKill detected (CRITICAL β€” immediate page) +- alert: OOMKillDetected + expr: | + increase(kube_pod_container_status_restarts_total{container="ghayma-api"}[5m]) > 0 + and kube_pod_container_status_last_terminated_reason{container="ghayma-api"} == "OOMKilled" + for: 0m + labels: + severity: critical + annotations: + summary: "OOMKill detected on Ghayma API" + description: "Pod {{ $labels.pod }} was OOMKilled. Immediate investigation required." + +# Alert: High error rate +- alert: HighErrorRate + expr: | + rate(http_requests_total{status=~"5.."}[5m]) + / rate(http_requests_total[5m]) > 0.05 + for: 3m + labels: + severity: critical + annotations: + summary: "Ghayma API error rate exceeds 5%" +``` + +### 7.3 Monitoring Architecture + +``` +β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” +β”‚ MONITORING STACK β”‚ +β”‚ β”‚ +β”‚ β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”‚ +β”‚ β”‚ Ghayma │───▢│ Prometheus │───▢│ Grafana Dashboard β”‚ β”‚ +β”‚ β”‚ API β”‚ β”‚ (scrapes β”‚ β”‚ (visualization + β”‚ β”‚ +β”‚ β”‚ /metrics β”‚ β”‚ /metrics) β”‚ β”‚ alerting rules) β”‚ β”‚ +β”‚ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β””β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”˜ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β”‚ +β”‚ β”‚ β”‚ +β”‚ β–Ό β”‚ +β”‚ β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”‚ +β”‚ β”‚ Alert Manager β”‚ β”‚ +β”‚ β”‚ (routing & β”‚ β”‚ +β”‚ β”‚ deduplication)β”‚ β”‚ +β”‚ β””β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β”‚ +β”‚ β”‚ β”‚ +β”‚ β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”‚ +β”‚ β–Ό β–Ό β–Ό β”‚ +β”‚ β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”‚ +β”‚ β”‚ Slack β”‚ β”‚ PagerDutyβ”‚ β”‚ Email β”‚ β”‚ +β”‚ β”‚ #alerts β”‚ β”‚ on-call β”‚ β”‚ team β”‚ β”‚ +β”‚ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β”‚ +β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ +``` + +### 7.4 Cloud Platform Native Tools + +| Cloud Platform | Monitoring Service | Key Feature for OOM Detection | +| :--- | :--- | :--- | +| **AWS ECS / Fargate** | CloudWatch Container Insights | `MemoryUtilization` metric + CloudWatch Alarms | +| **Google Cloud Run** | Cloud Monitoring | `container/memory/utilization` + Alert Policies | +| **Azure Container Apps** | Azure Monitor | `MemoryWorkingSet` + Metric Alerts | +| **Kubernetes (any)** | Prometheus + Grafana | `container_memory_usage_bytes` + `kube_pod_container_status_last_terminated_reason` | + +### 7.5 Recommended Grafana Dashboard Panels + +1. **Memory Usage vs Limit** β€” Time series showing memory consumption relative to the container limit (with 70% and 85% threshold lines) +2. **Container Restarts** β€” Counter panel showing restart events over time +3. **Response Time Heatmap** β€” p50 / p95 / p99 latency distribution +4. **Request Rate & Error Rate** β€” Dual-axis graph (total requests + 5xx rate) +5. **Pod/Replica Count** β€” Shows HPA scaling activity over time + +--- + +## 8. Lessons Learned + +| # | Lesson | Action Item | +| :--- | :--- | :--- | +| 1 | Default memory limits were set too low without load testing | Conduct load tests before production deployment | +| 2 | Single-replica deployment has no resilience | Always run **β‰₯ 2 replicas** for production services | +| 3 | Alert threshold (10 min) was too slow for a P1 outage | Reduce critical alert threshold to **2 minutes** | +| 4 | In-memory data stores are dangerous without bounds | Use external databases or enforce collection size limits | +| 5 | No runbook existed for OOMKilled incidents | Create and publish an OOMKilled response runbook | + +--- + +## 9. Action Items Tracker + +| # | Action Item | Owner | Priority | Deadline | Status | +| :--- | :--- | :--- | :--- | :--- | :--- | +| 1 | Increase container memory limit to 512MB | DevOps | P0 | Done | βœ… | +| 2 | Add `express.json({ limit: '1mb' })` payload limit | Backend | P1 | +1 day | ⬜ | +| 3 | Implement HPA with memory-based scaling | DevOps | P1 | +3 days | ⬜ | +| 4 | Set up Prometheus memory alerts (70% / 85%) | DevOps | P1 | +3 days | ⬜ | +| 5 | Add OOMKilled alert rule (immediate paging) | DevOps | P0 | +1 day | ⬜ | +| 6 | Run minimum 2 replicas in production | DevOps | P1 | +1 day | ⬜ | +| 7 | Replace in-memory store with Redis/DB | Backend | P2 | +1 week | ⬜ | +| 8 | Conduct load testing with realistic traffic | QA | P2 | +2 weeks | ⬜ | +| 9 | Create OOMKilled incident runbook | DevOps | P2 | +1 week | ⬜ | +| 10 | Build Grafana dashboard for container metrics | DevOps | P2 | +1 week | ⬜ | + +--- + +> **Report prepared by:** DevOps Team +> **Review date:** July 26, 2026 +> **Next review:** August 2, 2026 (verify all P0/P1 items completed) diff --git a/q3-cicd/workflow.yml b/q3-cicd/workflow.yml new file mode 100644 index 0000000..f952f1b --- /dev/null +++ b/q3-cicd/workflow.yml @@ -0,0 +1,218 @@ +name: CI/CD Pipeline β€” Build, Test & Deploy to Ghayma Cloud + +on: + push: + branches: [main] + pull_request: + branches: [main] + +env: + REGISTRY: registry.ghaymah.systems + IMAGE_NAME: ghayma-api + STAGING_APP: ghayma-api-staging + PRODUCTION_APP: ghayma-api-production + +jobs: + # ────────────────────────────────────────────── + # Job 1: Build & Test + # ────────────────────────────────────────────── + build-and-test: + name: πŸ”¨ Build & Test + runs-on: ubuntu-latest + + steps: + - name: Checkout code + uses: actions/checkout@v4 + + - name: Setup Node.js + uses: actions/setup-node@v4 + with: + node-version: '20' + cache: 'npm' + + - name: Install dependencies + run: npm ci + + - name: Run syntax check + run: node -c src/index.js + + - name: Run health check test + run: | + # Start the server in the background + node src/index.js & + SERVER_PID=$! + sleep 3 + + # Test health endpoint + HTTP_STATUS=$(curl -s -o /dev/null -w "%{http_code}" http://localhost:3000/health) + if [ "$HTTP_STATUS" -ne 200 ]; then + echo "❌ Health check failed with status $HTTP_STATUS" + kill $SERVER_PID + exit 1 + fi + + echo "βœ… Health check passed (HTTP $HTTP_STATUS)" + kill $SERVER_PID + + # ────────────────────────────────────────────── + # Job 2: Build Docker Image & Push to Registry + # ────────────────────────────────────────────── + docker-build-push: + name: 🐳 Build & Push Docker Image + runs-on: ubuntu-latest + needs: build-and-test + + outputs: + image-tag: ${{ steps.meta.outputs.tags }} + image-digest: ${{ steps.build-push.outputs.digest }} + + steps: + - name: Checkout code + uses: actions/checkout@v4 + + - name: Log in to Ghayma Container Registry + uses: docker/login-action@v3 + with: + registry: ${{ env.REGISTRY }} + username: ${{ secrets.GHAYMAH_REGISTRY_USERNAME }} + password: ${{ secrets.GHAYMAH_REGISTRY_PASSWORD }} + + - name: Extract Docker metadata (tags & labels) + id: meta + uses: docker/metadata-action@v5 + with: + images: ${{ env.REGISTRY }}/${{ secrets.GHAYMAH_REGISTRY_USERNAME }}/${{ env.IMAGE_NAME }} + tags: | + type=sha,prefix= + type=ref,event=branch + type=raw,value=latest,enable={{is_default_branch}} + + - name: Build and push Docker image + id: build-push + uses: docker/build-push-action@v5 + with: + context: . + push: true + tags: ${{ steps.meta.outputs.tags }} + labels: ${{ steps.meta.outputs.labels }} + + - name: Print image details + run: | + echo "βœ… Image pushed successfully" + echo "πŸ“¦ Tags: ${{ steps.meta.outputs.tags }}" + echo "πŸ”’ Digest: ${{ steps.build-push.outputs.digest }}" + + # ────────────────────────────────────────────── + # Job 3: Deploy to Staging (automatic) + # ────────────────────────────────────────────── + deploy-staging: + name: 🟑 Deploy to Staging + runs-on: ubuntu-latest + needs: docker-build-push + if: github.ref == 'refs/heads/main' + environment: + name: staging + url: https://staging.ghaymah.systems + + steps: + - name: Install Ghayma CLI + run: | + curl -fsSL https://cli.ghaymah.systems/install.sh | bash + echo "$HOME/.ghaymah/bin" >> $GITHUB_PATH + + - name: Authenticate with Ghayma Cloud + run: | + ghaymah auth login \ + --token ${{ secrets.GHAYMAH_API_TOKEN }} + + - name: Deploy to Staging environment + run: | + ghaymah app deploy ${{ env.STAGING_APP }} \ + --image ${{ env.REGISTRY }}/${{ secrets.GHAYMAH_REGISTRY_USERNAME }}/${{ env.IMAGE_NAME }}:latest \ + --env NODE_ENV=staging \ + --env PORT=3000 \ + --region me-central-1 \ + --wait + + - name: Verify Staging deployment + run: | + sleep 10 + HTTP_STATUS=$(curl -s -o /dev/null -w "%{http_code}" https://staging.ghaymah.systems/health) + if [ "$HTTP_STATUS" -ne 200 ]; then + echo "❌ Staging health check failed (HTTP $HTTP_STATUS)" + exit 1 + fi + echo "βœ… Staging deployment verified (HTTP $HTTP_STATUS)" + + # ────────────────────────────────────────────── + # Job 4: Manual Approval Gate + # ────────────────────────────────────────────── + # This job uses a GitHub Environment called + # "production-approval" with "Required reviewers" + # protection rule. The workflow PAUSES here and + # waits for a designated team member to approve. + # ────────────────────────────────────────────── + manual-approval: + name: ⏸️ Manual Approval Gate + runs-on: ubuntu-latest + needs: deploy-staging + if: github.ref == 'refs/heads/main' + environment: + name: production-approval # Configure "Required reviewers" on this environment + + steps: + - name: Approval checkpoint + run: | + echo "βœ… Production deployment has been approved!" + echo "πŸ‘€ Approved by: ${{ github.actor }}" + echo "πŸ“¦ Image: ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}:latest" + echo "⏰ Approved at: $(date -u '+%Y-%m-%d %H:%M:%S UTC')" + + # ────────────────────────────────────────────── + # Job 5: Deploy to Production (after approval) + # ────────────────────────────────────────────── + deploy-production: + name: 🟒 Deploy to Production + runs-on: ubuntu-latest + needs: manual-approval + if: github.ref == 'refs/heads/main' + environment: + name: production + url: https://api.ghaymah.systems + + steps: + - name: Install Ghayma CLI + run: | + curl -fsSL https://cli.ghaymah.systems/install.sh | bash + echo "$HOME/.ghaymah/bin" >> $GITHUB_PATH + + - name: Authenticate with Ghayma Cloud + run: | + ghaymah auth login \ + --token ${{ secrets.GHAYMAH_API_TOKEN }} + + - name: Deploy to Production environment + run: | + ghaymah app deploy ${{ env.PRODUCTION_APP }} \ + --image ${{ env.REGISTRY }}/${{ secrets.GHAYMAH_REGISTRY_USERNAME }}/${{ env.IMAGE_NAME }}:latest \ + --env NODE_ENV=production \ + --env PORT=3000 \ + --region me-central-1 \ + --replicas 2 \ + --wait + + - name: Verify Production deployment + run: | + sleep 10 + HTTP_STATUS=$(curl -s -o /dev/null -w "%{http_code}" https://api.ghaymah.systems/health) + if [ "$HTTP_STATUS" -ne 200 ]; then + echo "❌ Production health check failed (HTTP $HTTP_STATUS)" + exit 1 + fi + echo "βœ… Production deployment verified (HTTP $HTTP_STATUS)" + + - name: Notify success + run: | + echo "πŸš€ Production deployment complete!" + echo "πŸ“¦ Image: ${{ env.REGISTRY }}/${{ secrets.GHAYMAH_REGISTRY_USERNAME }}/${{ env.IMAGE_NAME }}:latest" + echo "🌐 URL: https://api.ghaymah.systems" diff --git a/q4-scalability/Ghaymah Scalability Diagram.png b/q4-scalability/Ghaymah Scalability Diagram.png new file mode 100644 index 0000000..6b486c5 Binary files /dev/null and b/q4-scalability/Ghaymah Scalability Diagram.png differ diff --git a/q4-scalability/calculations.md b/q4-scalability/calculations.md new file mode 100644 index 0000000..b4a46af --- /dev/null +++ b/q4-scalability/calculations.md @@ -0,0 +1,648 @@ +# Scalability & Load Distribution β€” Architecture Report + +**Project:** Ghayma REST API +**Platform:** ghaymah.systems (Ghayma Cloud) +**Target Load:** 15,000 requests/second +**Date:** July 26, 2026 + +--- + +## Table of Contents + +1. [Architecture Diagram β€” 15,000 req/s](#1-architecture-diagram--15000-reqs) +2. [Container Capacity Calculation](#2-container-capacity-calculation) +3. [Cold Start Strategy for New Containers](#3-cold-start-strategy-for-new-containers) +4. [Ghayma Block Storage for Stateful Workloads](#4-ghayma-block-storage-for-stateful-workloads) + +--- + +## 1. Architecture Diagram β€” 15,000 req/s + +### High-Level System Architecture + +``` + 15,000 req/s + β”‚ + β–Ό + β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” + β”‚ DNS / CDN β”‚ + β”‚ (dns.ghaymah.systems)β”‚ + β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ + β”‚ + β–Ό + β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” + β”‚ LOAD BALANCER β”‚ + β”‚ (Layer 7 - HTTPS) β”‚ + β”‚ β”‚ + β”‚ β€’ SSL Termination β”‚ + β”‚ β€’ Health Checks β”‚ + β”‚ β€’ Round Robin / β”‚ + β”‚ Least Connections β”‚ + β”‚ β€’ Rate Limiting β”‚ + β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ + β”‚ + β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” + β”‚ β”‚ β”‚ + β–Ό β–Ό β–Ό + β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” + β”‚ Zone A β”‚ β”‚ Zone B β”‚ β”‚ Zone C β”‚ + β”‚ (Rack/AZ 1) β”‚ β”‚ (Rack/AZ 2) β”‚ β”‚ (Rack/AZ 3) β”‚ + β””β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”˜ β””β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”˜ β””β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”˜ + β”‚ β”‚ β”‚ + β”Œβ”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β” β”Œβ”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β” β”Œβ”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β” + β”‚ β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”‚ β”‚ β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”‚ β”‚ β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”‚ + β”‚ β”‚Containerβ”‚ β”‚ β”‚ β”‚Containerβ”‚ β”‚ β”‚ β”‚Containerβ”‚ β”‚ + β”‚ β”‚ 1-15 β”‚ β”‚ β”‚ β”‚ 16-30 β”‚ β”‚ β”‚ β”‚ 31-44 β”‚ β”‚ + β”‚ β”‚ (15x) β”‚ β”‚ β”‚ β”‚ (15x) β”‚ β”‚ β”‚ β”‚ (14x) β”‚ β”‚ + β”‚ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β”‚ β”‚ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β”‚ β”‚ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β”‚ + β””β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”˜ β””β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”˜ β””β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”˜ + β”‚ β”‚ β”‚ + β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ + β”‚ + β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” + β”‚ β”‚ β”‚ + β–Ό β–Ό β–Ό + β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” + β”‚ Database β”‚ β”‚ Redis β”‚ β”‚ Block Storageβ”‚ + β”‚ (Primary + β”‚ β”‚ (Session & β”‚ β”‚ (Persistent β”‚ + β”‚ Replicas) β”‚ β”‚ Cache) β”‚ β”‚ Volumes) β”‚ + β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ +``` + +### Detailed Component Architecture + +``` +β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” +β”‚ β”‚ +β”‚ GHAYMA CLOUD PLATFORM β”‚ +β”‚ β”‚ +β”‚ β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”‚ +β”‚ β”‚ NETWORKING LAYER β”‚ β”‚ +β”‚ β”‚ β”‚ β”‚ +β”‚ β”‚ Internet ──▢ [DNS] ──▢ [CDN/WAF] ──▢ [Load Balancer (L7)] β”‚ β”‚ +β”‚ β”‚ β”‚ β”‚ β”‚ +β”‚ β”‚ β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€ β”‚ β”‚ +β”‚ β”‚ β”‚ Health Checks β”‚ β”‚ β”‚ +β”‚ β”‚ β”‚ GET /health (10s) β”‚ β”‚ β”‚ +β”‚ β”‚ β”‚ Threshold: 3 fails β”‚ β”‚ β”‚ +β”‚ β”‚ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β”‚ β”‚ +β”‚ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β”‚ +β”‚ β”‚ β”‚ +β”‚ β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”‚ +β”‚ β”‚ COMPUTE LAYER (44 Containers) β”‚ β”‚ +β”‚ β”‚ β”‚ β”‚ +β”‚ β”‚ β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”‚ β”‚ +β”‚ β”‚ β”‚ C-01 β”‚ β”‚ C-02 β”‚ β”‚ C-03 β”‚ β”‚ C-04 β”‚ ... β”‚ C-44 β”‚ β”‚ β”‚ +β”‚ β”‚ β”‚ 500r/s β”‚ β”‚ 500r/s β”‚ β”‚ 500r/s β”‚ β”‚ 500r/s β”‚ β”‚ 500r/s β”‚ β”‚ β”‚ +β”‚ β”‚ β”‚ 512MB β”‚ β”‚ 512MB β”‚ β”‚ 512MB β”‚ β”‚ 512MB β”‚ β”‚ 512MB β”‚ β”‚ β”‚ +β”‚ β”‚ β”‚ 0.5 CPU β”‚ β”‚ 0.5 CPU β”‚ β”‚ 0.5 CPU β”‚ β”‚ 0.5 CPU β”‚ β”‚ 0.5 CPU β”‚ β”‚ β”‚ +β”‚ β”‚ β””β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”˜ β””β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”˜ β””β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”˜ β””β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”˜ β””β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”˜ β”‚ β”‚ +β”‚ β”‚ └────────────┴────────────┴────────────┴──── ... β”€β”€β”€β”€β”€β”€β”€β”˜ β”‚ β”‚ +β”‚ β”‚ β”‚ β”‚ β”‚ +β”‚ β”‚ β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”‚ β”‚ +β”‚ β”‚ β”‚ AUTO-SCALER (HPA) β”‚ β”‚ β”‚ +β”‚ β”‚ β”‚ Min: 44 | Max: 66 | Scale at: CPU>75% or Mem>70% β”‚ β”‚ β”‚ +β”‚ β”‚ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β”‚ β”‚ +β”‚ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β”‚ +β”‚ β”‚ β”‚ +β”‚ β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”‚ +β”‚ β”‚ DATA LAYER β”‚ β”‚ +β”‚ β”‚ β”‚ β”‚ +β”‚ β”‚ β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”‚ β”‚ +β”‚ β”‚ β”‚ PostgreSQL DB β”‚ β”‚ Redis β”‚ β”‚ Ghayma Block Storage β”‚ β”‚ β”‚ +β”‚ β”‚ β”‚ Primary + 2 β”‚ β”‚ Cluster β”‚ β”‚ (Persistent Volumes) β”‚ β”‚ β”‚ +β”‚ β”‚ β”‚ Read Replicas β”‚ β”‚ (Cache + β”‚ β”‚ β€’ Database data (100GB) β”‚ β”‚ β”‚ +β”‚ β”‚ β”‚ β”‚ β”‚ Sessions) β”‚ β”‚ β€’ Logs & uploads (50GB) β”‚ β”‚ β”‚ +β”‚ β”‚ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β”‚ β”‚ +β”‚ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β”‚ +β”‚ β”‚ +β”‚ β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”‚ +β”‚ β”‚ OBSERVABILITY LAYER β”‚ β”‚ +β”‚ β”‚ β”‚ β”‚ +β”‚ β”‚ [Prometheus] ──▢ [Grafana Dashboards] ──▢ [AlertManager] ──▢ [PagerDuty] β”‚ β”‚ +β”‚ β”‚ [Container Logs] ──▢ [Centralized Logging] ──▢ [Log Analytics] β”‚ β”‚ +β”‚ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β”‚ +β”‚ β”‚ +β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ +``` + +### Load Balancer Distribution Strategy + +``` + 15,000 req/s incoming + β”‚ + β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β” + β”‚ LOAD BALANCER β”‚ + β”‚ β”‚ + β”‚ Algorithm: β”‚ + β”‚ Least β”‚ + β”‚ Connections β”‚ + β””β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”˜ + β”‚ + β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” + β”‚ β”‚ β”‚ + Zone A (33%) Zone B (33%) Zone C (34%) + ~5,000 r/s ~5,000 r/s ~5,000 r/s + 15 containers 15 containers 14 containers + ~333 r/s each ~333 r/s each ~357 r/s each +``` + +> Each container handles **~341 req/s on average** β€” well within the 500 req/s capacity, leaving the 30% safety margin intact. + +--- + +## 2. Container Capacity Calculation + +### Given Parameters + +| Parameter | Value | +| :--- | :--- | +| **Total traffic** | 15,000 req/s | +| **Capacity per container** | 500 req/s | +| **Safety margin** | 30% | + +### Step-by-Step Calculation + +#### Step 1: Calculate Effective Capacity per Container + +The 30% margin means each container should only be utilized at 70% of its max capacity: + +``` +Effective capacity = Max capacity Γ— (1 - Safety margin) +Effective capacity = 500 req/s Γ— (1 - 0.30) +Effective capacity = 500 Γ— 0.70 +Effective capacity = 350 req/s per container +``` + +#### Step 2: Calculate Minimum Containers Required + +``` +Minimum containers = Total traffic Γ· Effective capacity +Minimum containers = 15,000 Γ· 350 +Minimum containers = 42.86 +Minimum containers = 43 containers (rounded up) +``` + +#### Step 3: Add Availability Buffer (+1) + +To handle single-container failures without breaching capacity: + +``` +Recommended containers = 43 + 1 (failure tolerance) +Recommended containers = 44 containers +``` + +### Summary Table + +| Metric | Value | Formula | +| :--- | :--- | :--- | +| **Total load** | 15,000 req/s | Given | +| **Max capacity/container** | 500 req/s | Given | +| **Safety margin** | 30% | Given | +| **Effective capacity/container** | 350 req/s | 500 Γ— 0.70 | +| **Minimum containers** | 43 | ⌈15,000 Γ· 350βŒ‰ | +| **Recommended (with buffer)** | **44** | 43 + 1 | +| **Actual utilization per container** | ~341 req/s | 15,000 Γ· 44 | +| **Utilization percentage** | 68.2% | 341 Γ· 500 | +| **Remaining headroom** | 31.8% | βœ… Exceeds 30% margin | + +### Capacity Planning Visualization + +``` +Container Capacity: 500 req/s + β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€ + 0 500 + +Actual load per container: ~341 req/s (68.2%) + β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€ + 0 341 + +Safety margin zone (30%): + β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β–‘β–‘β–‘β–‘β–‘β–‘β–‘β–‘β–‘β–‘β–‘β–‘β–‘β–‘β”€ + 0 350 500 + β–² + Threshold + (don't exceed) +``` + +### Auto-Scaling Tiers + +| Traffic Level | Containers Needed | Status | +| :--- | :--- | :--- | +| 5,000 req/s (low) | 15 | Minimum viable | +| 10,000 req/s (moderate) | 30 | Normal scaling | +| **15,000 req/s (target)** | **44** | **Design target** | +| 20,000 req/s (spike) | 58 | Burst capacity | +| 25,000 req/s (peak) | 72 | Maximum scaling | + +### Resource Allocation per Container + +| Resource | Value | +| :--- | :--- | +| Memory Request | 256 Mi | +| Memory Limit | 512 Mi | +| CPU Request | 250m (0.25 vCPU) | +| CPU Limit | 500m (0.5 vCPU) | + +### Total Cluster Resources (44 containers) + +| Resource | Per Container | Γ— 44 Containers | Total | +| :--- | :--- | :--- | :--- | +| **Memory** | 512 Mi | Γ— 44 | **22 GB** | +| **CPU** | 0.5 vCPU | Γ— 44 | **22 vCPUs** | +| **Throughput** | 350 req/s (effective) | Γ— 44 | **15,400 req/s** | + +--- + +## 3. Cold Start Strategy for New Containers + +### What is a Cold Start? + +A **cold start** occurs when a new container instance is created and must initialize before serving traffic. During this period: + +- The container image is pulled from the registry +- The Node.js runtime starts +- Dependencies are loaded +- The application connects to databases/caches +- The first requests experience **higher latency** + +``` + Cold Start Timeline (without optimization) + ────────────────────────────────────────────────────── + β”‚ Image Pull β”‚ Runtime Init β”‚ App Boot β”‚ DB Connect β”‚ Ready β”‚ + 0s 3s 4s 5s 7s 8s + β–² + First request served +``` + +### Cold Start Strategy β€” 5 Layers + +#### Layer 1: Pre-Warmed Container Pool + +Maintain a pool of **idle but ready** containers that can immediately accept traffic: + +``` +Active Containers (44) Pre-Warmed Pool (4-6) +β”Œβ”€β”€β”€β”€β”¬β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”¬β”€ ... ─┬────┐ β”Œβ”€β”€β”€β”€β”¬β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”¬β”€β”€β”€β”€β” +β”‚ C1 β”‚ C2 β”‚ C3 β”‚ β”‚C44 β”‚ β”‚ W1 β”‚ W2 β”‚ W3 β”‚ W4 β”‚ +β”‚BUSYβ”‚BUSYβ”‚BUSYβ”‚ β”‚BUSYβ”‚ β”‚IDLEβ”‚IDLEβ”‚IDLEβ”‚IDLEβ”‚ +└────┴────┴────┴─ ... β”€β”΄β”€β”€β”€β”€β”˜ β””β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”˜ + β”‚ + Traffic spike detected + β”‚ + β–Ό + W1, W2 immediately start + serving (0s cold start) +``` + +**Implementation:** +```yaml +# HPA with pre-warmed minimum +apiVersion: autoscaling/v2 +kind: HorizontalPodAutoscaler +spec: + minReplicas: 48 # 44 active + 4 pre-warmed + maxReplicas: 72 +``` + +#### Layer 2: Optimized Docker Image (Fast Pulls) + +Reduce image pull time by using lightweight images: + +```dockerfile +# βœ… GOOD: Alpine-based image (~150MB, pulls in ~2s) +FROM node:20-alpine + +# ❌ BAD: Full Debian image (~1GB, pulls in ~12s) +FROM node:20 +``` + +| Image Base | Size | Pull Time | Cold Start Impact | +| :--- | :--- | :--- | :--- | +| `node:20` (Debian) | ~1 GB | ~12s | Very slow | +| `node:20-slim` | ~250 MB | ~5s | Moderate | +| `node:20-alpine` | ~150 MB | ~2s | βœ… **Fast** | + +#### Layer 3: Application-Level Readiness + +Implement a **readiness probe** that only marks the container as "ready" after it has fully initialized: + +```javascript +// In src/index.js +let isReady = false; + +// Readiness check - only returns 200 when app is fully initialized +app.get('/ready', (req, res) => { + if (isReady) { + res.status(200).json({ ready: true }); + } else { + res.status(503).json({ ready: false, message: 'Warming up...' }); + } +}); + +// App startup sequence +async function initialize() { + // 1. Connect to database + await connectDatabase(); + + // 2. Connect to Redis cache + await connectRedis(); + + // 3. Pre-load frequently accessed data into memory + await preloadCache(); + + // 4. Mark as ready ONLY after everything is initialized + isReady = true; + console.log('βœ… Application fully initialized and ready'); +} + +app.listen(PORT, async () => { + console.log(`Server starting on port ${PORT}...`); + await initialize(); +}); +``` + +```yaml +# Kubernetes readiness probe +readinessProbe: + httpGet: + path: /ready # Not /health β€” separate from liveness + port: 3000 + initialDelaySeconds: 3 + periodSeconds: 5 + failureThreshold: 3 # Must pass 3 checks before receiving traffic +``` + +> The load balancer **will not route traffic** to the container until `/ready` returns HTTP 200. + +#### Layer 4: Proactive Scaling (Anticipate Demand) + +Scale **before** the spike hits, not after: + +```yaml +# CronJob-based pre-scaling for known traffic patterns +apiVersion: batch/v1 +kind: CronJob +metadata: + name: pre-scale-morning +spec: + # Scale up at 8:00 AM before morning traffic rush + schedule: "0 8 * * 1-5" + jobTemplate: + spec: + template: + spec: + containers: + - name: scaler + image: bitnami/kubectl:latest + command: + - kubectl + - scale + - deployment/ghayma-api + - --replicas=50 +``` + +#### Layer 5: Connection Pooling & Warm-Up Requests + +After a container starts, send **internal warm-up requests** to pre-establish connections: + +```javascript +async function warmUp() { + console.log('πŸ”₯ Running warm-up sequence...'); + + // Pre-establish database connection pool + await db.query('SELECT 1'); + + // Prime V8 JIT compiler with typical request patterns + const warmUpPaths = ['/health', '/api/v1/items']; + for (const path of warmUpPaths) { + await fetch(`http://localhost:${PORT}${path}`); + } + + console.log('βœ… Warm-up complete β€” ready for production traffic'); +} +``` + +### Cold Start Strategy Summary + +``` +β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” +β”‚ COLD START MITIGATION LAYERS β”‚ +β”‚ β”‚ +β”‚ Layer 1: Pre-Warmed Pool β†’ 0s cold start (instant) β”‚ +β”‚ Layer 2: Alpine Docker Image β†’ 2s image pull (vs 12s) β”‚ +β”‚ Layer 3: Readiness Probes β†’ No traffic until fully ready β”‚ +β”‚ Layer 4: Proactive Scaling β†’ Scale before the spike β”‚ +β”‚ Layer 5: Warm-Up Requests β†’ Pre-establish connections β”‚ +β”‚ β”‚ +β”‚ Combined Effect: New containers ready in ~3-5s β”‚ +β”‚ Without strategy: New containers ready in ~15-20s β”‚ +β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ +``` + +--- + +## 4. Ghayma Block Storage for Stateful Workloads + +### What is Ghayma Block Storage? + +**Ghayma Block Storage** is a persistent, network-attached block storage service provided by the Ghayma Cloud platform. It provides raw storage volumes that can be attached to containers, similar to a virtual hard drive. + +``` +β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” +β”‚ BLOCK STORAGE vs CONTAINER STORAGE β”‚ +β”‚ β”‚ +β”‚ Container (Ephemeral) Block Storage (Persistent) β”‚ +β”‚ β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”‚ +β”‚ β”‚ Container FS β”‚ β”‚ Ghayma Block Volume β”‚ β”‚ +β”‚ β”‚ β”‚ β”‚ β”‚ β”‚ +β”‚ β”‚ ❌ Data lost β”‚ β”‚ βœ… Data survives β”‚ β”‚ +β”‚ β”‚ when containerβ”‚ β”‚ container restartsβ”‚ β”‚ +β”‚ β”‚ stops/crashes β”‚ β”‚ and redeployments β”‚ β”‚ +β”‚ β”‚ β”‚ β”‚ β”‚ β”‚ +β”‚ β”‚ ❌ Not shared β”‚ β”‚ βœ… Can be reattached β”‚ β”‚ +β”‚ β”‚ between β”‚ β”‚ to new containers β”‚ β”‚ +β”‚ β”‚ containers β”‚ β”‚ β”‚ β”‚ +β”‚ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β”‚ +β”‚ β”‚ +β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ +``` + +### Why is Block Storage Needed? + +Containers are **stateless and ephemeral** by design β€” when a container restarts, crashes, or is replaced during a deployment, all data written inside it is **lost**. This is a problem for: + +| Workload Type | Data at Risk | Block Storage Solution | +| :--- | :--- | :--- | +| **Databases** (PostgreSQL, MySQL) | Table data, indexes, WAL logs | Persistent volume for `/var/lib/postgresql/data` | +| **Cache** (Redis with persistence) | RDB/AOF snapshots | Volume for `/data` | +| **File Uploads** | User-uploaded images, documents | Volume for `/app/uploads` | +| **Application Logs** | Log files for audit/compliance | Volume for `/var/log/app` | +| **Session Storage** | User session files | Volume for `/tmp/sessions` | + +### Architecture: Block Storage with Containers + +``` +β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” +β”‚ STATEFUL WORKLOAD ARCHITECTURE β”‚ +β”‚ β”‚ +β”‚ β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”‚ +β”‚ β”‚ API Container β”‚ β”‚ Database Container β”‚ β”‚ +β”‚ β”‚ (Stateless) β”‚ β”‚ (Stateful β€” PostgreSQL) β”‚ β”‚ +β”‚ β”‚ │──────▢│ β”‚ β”‚ +β”‚ β”‚ No volume β”‚ β”‚ β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”‚ β”‚ +β”‚ β”‚ needed β”‚ β”‚ β”‚ /var/lib/postgresql β”‚ β”‚ β”‚ +β”‚ β”‚ β”‚ β”‚ β”‚ (mount point) β”‚ β”‚ β”‚ +β”‚ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β”‚ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β”‚ β”‚ +β”‚ β”‚ β”‚ β”‚ β”‚ +β”‚ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β”‚ +β”‚ β”‚ β”‚ +β”‚ β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β–Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”‚ +β”‚ β”‚ GHAYMA BLOCK β”‚ β”‚ +β”‚ β”‚ STORAGE VOLUME β”‚ β”‚ +β”‚ β”‚ β”‚ β”‚ +β”‚ β”‚ Name: db-data β”‚ β”‚ +β”‚ β”‚ Size: 100 GB β”‚ β”‚ +β”‚ β”‚ Type: SSD (gp3) β”‚ β”‚ +β”‚ β”‚ IOPS: 3,000 β”‚ β”‚ +β”‚ β”‚ Encrypted: Yes β”‚ β”‚ +β”‚ β”‚ β”‚ β”‚ +β”‚ β”‚ βœ… Persists across β”‚ β”‚ +β”‚ β”‚ restarts β”‚ β”‚ +β”‚ β”‚ βœ… Automatic β”‚ β”‚ +β”‚ β”‚ snapshots β”‚ β”‚ +β”‚ β”‚ βœ… Reattachable β”‚ β”‚ +β”‚ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β”‚ +β”‚ β”‚ +β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ +``` + +### How to Use Ghayma Block Storage + +#### Step 1: Create a Block Storage Volume + +```bash +# Using Ghayma CLI +ghaymah storage create db-data \ + --size 100GB \ + --type ssd \ + --region me-central-1 \ + --encrypted +``` + +#### Step 2: Attach to a Container / Deployment + +```yaml +# Kubernetes PersistentVolumeClaim using Ghayma storage class +apiVersion: v1 +kind: PersistentVolumeClaim +metadata: + name: db-data-pvc +spec: + accessModes: + - ReadWriteOnce # Single container read/write + storageClassName: ghaymah-block-ssd + resources: + requests: + storage: 100Gi +--- +# Deployment using the volume +apiVersion: apps/v1 +kind: Deployment +metadata: + name: postgres-db +spec: + replicas: 1 # Stateful β€” typically 1 primary + template: + spec: + containers: + - name: postgres + image: postgres:16-alpine + ports: + - containerPort: 5432 + volumeMounts: + - name: db-storage + mountPath: /var/lib/postgresql/data + env: + - name: POSTGRES_PASSWORD + valueFrom: + secretKeyRef: + name: db-secret + key: password + volumes: + - name: db-storage + persistentVolumeClaim: + claimName: db-data-pvc # Ghayma Block Storage volume +``` + +#### Step 3: Verify the Volume + +```bash +# Check volume status +ghaymah storage list + +# Output: +# NAME SIZE TYPE STATUS ATTACHED TO REGION +# db-data 100GB SSD attached postgres-db-pod-0 me-central-1 +``` + +### Block Storage Features on Ghayma Cloud + +| Feature | Description | +| :--- | :--- | +| **Persistence** | Data survives container restarts, crashes, and redeployments | +| **SSD Performance** | Low-latency SSD storage with up to 16,000 IOPS | +| **Encryption** | At-rest encryption (AES-256) for security compliance | +| **Snapshots** | Point-in-time snapshots for backup and disaster recovery | +| **Resizing** | Expand volume size without downtime | +| **Reattachment** | Detach from one container and attach to another (for migration) | +| **Availability Zones** | Volume and container must be in the same zone | + +### When to Use Block Storage vs Other Options + +| Storage Type | Use Case | Persistence | Shared Access | Speed | +| :--- | :--- | :--- | :--- | :--- | +| **Container FS** | Temp files, build artifacts | ❌ Ephemeral | ❌ No | ⚑ Fastest | +| **Ghayma Block Storage** | Databases, stateful apps, logs | βœ… Persistent | ❌ Single container | ⚑ Fast (SSD) | +| **Object Storage (S3-like)** | Images, videos, backups | βœ… Persistent | βœ… Multi-container | 🐌 Moderate | +| **Shared File Storage (NFS)** | Shared config, media libraries | βœ… Persistent | βœ… Multi-container | πŸ”„ Moderate | + +### Data Lifecycle with Block Storage + +``` + Container crashes or redeploys + β”‚ + β–Ό + β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” + β”‚ Old container is terminated β”‚ + β”‚ ❌ Container filesystem is destroyed β”‚ + β”‚ βœ… Block Storage volume is PRESERVED β”‚ + β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ + β”‚ + β–Ό + β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” + β”‚ New container starts β”‚ + β”‚ Block Storage volume is reattached β”‚ + β”‚ Data is immediately available at the β”‚ + β”‚ same mount path (/var/lib/postgresql) β”‚ + β”‚ β”‚ + β”‚ βœ… Zero data loss β”‚ + β”‚ βœ… Database resumes from exact state β”‚ + β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ +``` + +--- + +## Summary + +| Requirement | Solution | +| :--- | :--- | +| **Architecture for 15K req/s** | 3-zone architecture with L7 load balancer, 44 containers, auto-scaler, and data layer | +| **Container count** | **44 containers** (15,000 Γ· 350 effective req/s + 1 buffer) at 68.2% utilization | +| **Cold start strategy** | 5-layer approach: pre-warmed pool, Alpine images, readiness probes, proactive scaling, warm-up requests | +| **Block Storage** | Persistent SSD volumes for databases and stateful workloads that survive container restarts | + +--- + +> **Document prepared for:** Ghayma Assessment β€” Scalability & Load Distribution section +> **Last updated:** July 26, 2026 diff --git a/q5-mithal-monitor/dashboard.html b/q5-mithal-monitor/dashboard.html new file mode 100644 index 0000000..a6307b7 --- /dev/null +++ b/q5-mithal-monitor/dashboard.html @@ -0,0 +1,678 @@ + + + + + + mithal.space β€” Monitoring Dashboard + + + + + +
+ + +
+
+ +

mithal.space Monitoring Dashboard

+
+
+
+
+ LOADING... +
+
Updated: β€”
+
+
+ + +
+ + +
+
+
🟒
+
Uptime (24h)
+
β€”%
+
+
0 / 0 checks passed
+
+ +
+
⚑
+
Avg Latency
+
β€” ms
+
Last hour
+
+ +
+
🌐
+
DNS Resolution
+
β€” ms
+
β€”
+
+ +
+
πŸ”’
+
SSL Certificate
+
β€” days
+
β€”
+
+ +
+
πŸ”
+
Search Response
+
β€” ms
+
β€”
+
+
+ + +
+
+
+
πŸ“ˆ Response Time β€” Last Hour
+
+
+
+ +
+
+
+ +
+
+
πŸ”’ SSL Certificate Details
+
+
+
+
+
β€”
+
Days Until Expiration
+
+
+ Status + β€” +
+
+ Issuer + β€” +
+
+ Expires On + β€” +
+
+
+
+
+ + +
+
+
+
πŸ“‹ Last 10 Test Results
+
+
+
+ + + + + + + + + + + + + + + +
TimeStatusHTTPLatencyDNSSearchSSL
Waiting for data from collector...
+
+
+
+
+ +
+
+ + + + + + + diff --git a/q5-mithal-monitor/monitor.py b/q5-mithal-monitor/monitor.py new file mode 100644 index 0000000..c12f78e --- /dev/null +++ b/q5-mithal-monitor/monitor.py @@ -0,0 +1,275 @@ +#!/usr/bin/env python3 +""" +Monitoring Collector for mithal.space +Collects metrics every 60 seconds: Latency, Uptime, SSL, DNS, Search Response. +Stores results in data/mithal-metrics.json. +""" + +import urllib.request +import urllib.error +import ssl +import socket +import json +import time +import datetime +import os +import sys + +# ───────────────────────────────────────────── +# Configuration +# ───────────────────────────────────────────── +TARGET_URL = "https://mithal.space" +SEARCH_URL = "https://mithal.space" # Search/query endpoint (adjust if different) +CHECK_INTERVAL = 60 # seconds (1 minute) +DATA_DIR = os.path.join(os.path.dirname(os.path.abspath(__file__)), "data") +DATA_FILE = os.path.join(DATA_DIR, "mithal-metrics.json") +MAX_RECORDS = 1440 # Keep 24 hours of data (1 per minute) +TIMEOUT = 10 # seconds + +# ANSI colors +GREEN = "\033[92m" +RED = "\033[91m" +YELLOW = "\033[93m" +CYAN = "\033[96m" +BOLD = "\033[1m" +RESET = "\033[0m" + + +def ensure_data_dir(): + """Create data directory if it doesn't exist.""" + os.makedirs(DATA_DIR, exist_ok=True) + if not os.path.exists(DATA_FILE): + with open(DATA_FILE, "w", encoding="utf-8") as f: + json.dump({"records": []}, f) + + +def load_data(): + """Load existing metrics data.""" + try: + with open(DATA_FILE, "r", encoding="utf-8") as f: + return json.load(f) + except (json.JSONDecodeError, FileNotFoundError): + return {"records": []} + + +def save_data(data): + """Save metrics data, keeping only the last MAX_RECORDS entries.""" + data["records"] = data["records"][-MAX_RECORDS:] + with open(DATA_FILE, "w", encoding="utf-8") as f: + json.dump(data, f, indent=2, ensure_ascii=False) + + +# ───────────────────────────────────────────── +# Metric Collectors +# ───────────────────────────────────────────── + +def check_latency(url): + """Measure HTTP response time (latency) in milliseconds.""" + try: + start = time.time() + req = urllib.request.Request(url, method="GET") + req.add_header("User-Agent", "GhaymaMonitor/1.0") + response = urllib.request.urlopen(req, timeout=TIMEOUT) + latency_ms = round((time.time() - start) * 1000) + status_code = response.status + response.close() + return { + "latency_ms": latency_ms, + "status_code": status_code, + "is_up": 200 <= status_code < 400, + "error": None + } + except urllib.error.HTTPError as e: + latency_ms = round((time.time() - start) * 1000) + return { + "latency_ms": latency_ms, + "status_code": e.code, + "is_up": False, + "error": f"HTTP {e.code}: {e.reason}" + } + except Exception as e: + return { + "latency_ms": None, + "status_code": None, + "is_up": False, + "error": str(e) + } + + +def check_ssl(hostname): + """Check SSL certificate status and expiration date.""" + try: + context = ssl.create_default_context() + with socket.create_connection((hostname, 443), timeout=TIMEOUT) as sock: + with context.wrap_socket(sock, server_hostname=hostname) as ssock: + cert = ssock.getpeercert() + + # Parse expiration date + expire_str = cert.get("notAfter", "") + expire_date = datetime.datetime.strptime( + expire_str, "%b %d %H:%M:%S %Y %Z" + ) + now = datetime.datetime.utcnow() + days_remaining = (expire_date - now).days + + # Get issuer + issuer_parts = dict(x[0] for x in cert.get("issuer", [])) + issuer = issuer_parts.get("organizationName", "Unknown") + + return { + "valid": True, + "issuer": issuer, + "expires": expire_date.strftime("%Y-%m-%d %H:%M:%S UTC"), + "days_remaining": days_remaining, + "error": None + } + except Exception as e: + return { + "valid": False, + "issuer": None, + "expires": None, + "days_remaining": None, + "error": str(e) + } + + +def check_dns(hostname): + """Measure DNS resolution time in milliseconds.""" + try: + start = time.time() + ip_address = socket.gethostbyname(hostname) + dns_time_ms = round((time.time() - start) * 1000) + return { + "dns_time_ms": dns_time_ms, + "resolved_ip": ip_address, + "error": None + } + except Exception as e: + return { + "dns_time_ms": None, + "resolved_ip": None, + "error": str(e) + } + + +def check_search_response(url): + """Send a search/query request and measure response time.""" + try: + search_url = url.rstrip("/") + "/" + start = time.time() + req = urllib.request.Request(search_url, method="GET") + req.add_header("User-Agent", "GhaymaMonitor/1.0") + response = urllib.request.urlopen(req, timeout=TIMEOUT) + search_time_ms = round((time.time() - start) * 1000) + content_length = len(response.read()) + response.close() + return { + "search_time_ms": search_time_ms, + "content_length": content_length, + "status_code": response.status, + "error": None + } + except Exception as e: + return { + "search_time_ms": None, + "content_length": None, + "status_code": None, + "error": str(e) + } + + +# ───────────────────────────────────────────── +# Main Collector +# ───────────────────────────────────────────── + +def collect_metrics(): + """Run all metric checks and return a combined record.""" + timestamp = datetime.datetime.utcnow().strftime("%Y-%m-%dT%H:%M:%SZ") + hostname = TARGET_URL.replace("https://", "").replace("http://", "").split("/")[0] + + print(f"\n{CYAN}[{timestamp}] Collecting metrics for {TARGET_URL}...{RESET}") + + # 1. DNS Resolution + dns_result = check_dns(hostname) + status = f"{GREEN}βœ… {dns_result['dns_time_ms']}ms{RESET}" if not dns_result["error"] else f"{RED}❌ {dns_result['error']}{RESET}" + print(f" DNS Resolution: {status}") + + # 2. HTTP Latency & Uptime + latency_result = check_latency(TARGET_URL) + if latency_result["is_up"]: + status = f"{GREEN}βœ… {latency_result['status_code']} in {latency_result['latency_ms']}ms{RESET}" + else: + status = f"{RED}❌ {latency_result.get('error', 'Down')}{RESET}" + print(f" HTTP Latency: {status}") + + # 3. SSL Certificate + ssl_result = check_ssl(hostname) + if ssl_result["valid"]: + days = ssl_result["days_remaining"] + color = GREEN if days > 30 else (YELLOW if days > 7 else RED) + status = f"{color}βœ… Valid, {days} days remaining{RESET}" + else: + status = f"{RED}❌ {ssl_result.get('error', 'Invalid')}{RESET}" + print(f" SSL Status: {status}") + + # 4. Search Response + search_result = check_search_response(SEARCH_URL) + if search_result["search_time_ms"]: + status = f"{GREEN}βœ… {search_result['search_time_ms']}ms ({search_result['content_length']} bytes){RESET}" + else: + status = f"{RED}❌ {search_result.get('error', 'Failed')}{RESET}" + print(f" Search Response: {status}") + + # Combine into a single record + record = { + "timestamp": timestamp, + "latency": latency_result, + "uptime": { + "is_up": latency_result["is_up"], + "status_code": latency_result["status_code"] + }, + "ssl": ssl_result, + "dns": dns_result, + "search": search_result + } + + return record + + +def main(): + ensure_data_dir() + + print(f"{BOLD}{CYAN}") + print("=" * 58) + print(" mithal.space β€” Monitoring Collector") + print(f" Target: {TARGET_URL}") + print(f" Interval: {CHECK_INTERVAL}s (every minute)") + print(f" Storage: {DATA_FILE}") + print("=" * 58) + print(f"{RESET}") + + try: + while True: + record = collect_metrics() + + # Load, append, save + data = load_data() + data["records"].append(record) + save_data(data) + + up_count = sum(1 for r in data["records"] if r["uptime"]["is_up"]) + total = len(data["records"]) + uptime_pct = round((up_count / total) * 100, 2) if total > 0 else 0 + + print(f" {BOLD}Uptime (session): {uptime_pct}% ({up_count}/{total} checks){RESET}") + print(f" {CYAN}Next check in {CHECK_INTERVAL}s...{RESET}") + + time.sleep(CHECK_INTERVAL) + + except KeyboardInterrupt: + print(f"\n{YELLOW}Collector stopped.{RESET}") + sys.exit(0) + + +if __name__ == "__main__": + main()