feat: Ghayma Assessment - complete submission
This commit is contained in:
@@ -0,0 +1,77 @@
|
||||
# Ghayma REST API Application
|
||||
|
||||
A simple, lightweight Node.js Express REST API application with container support.
|
||||
|
||||
## 🚀 Features
|
||||
|
||||
- **Express.js** lightweight web framework
|
||||
- **CORS & JSON Middleware** pre-configured
|
||||
- **Health Check Endpoint** (`/health`) for Docker/Kubernetes probes
|
||||
- **Resource CRUD API** (`/api/v1/items`)
|
||||
- **Docker Support** with health checks and optimized `.dockerignore`
|
||||
|
||||
---
|
||||
|
||||
## 🛠️ Endpoints Overview
|
||||
|
||||
| Method | Endpoint | Description |
|
||||
| :--- | :--- | :--- |
|
||||
| `GET` | `/` | API Metadata & Welcome Info |
|
||||
| `GET` | `/health` | Application Health & Uptime Status |
|
||||
| `GET` | `/api/v1/items` | List all items |
|
||||
| `GET` | `/api/v1/items/:id` | Get item by ID |
|
||||
| `POST` | `/api/v1/items` | Create new item |
|
||||
| `DELETE` | `/api/v1/items/:id` | Delete item by ID |
|
||||
|
||||
---
|
||||
|
||||
## 💻 Running Locally
|
||||
|
||||
### Prerequisites
|
||||
- Node.js (v18+)
|
||||
|
||||
### Steps
|
||||
1. Install dependencies:
|
||||
```bash
|
||||
npm install
|
||||
```
|
||||
|
||||
2. Start the API server:
|
||||
```bash
|
||||
npm start
|
||||
```
|
||||
Or for development auto-reload (Node 18+):
|
||||
```bash
|
||||
npm run dev
|
||||
```
|
||||
|
||||
3. Test the server:
|
||||
Navigate to `http://localhost:3000` or run:
|
||||
```bash
|
||||
curl http://localhost:3000/health
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 🐳 Running with Docker
|
||||
|
||||
### Build the Docker Image
|
||||
```bash
|
||||
docker build -t ghayma-api .
|
||||
```
|
||||
|
||||
### Run the Container
|
||||
```bash
|
||||
docker run -d -p 3000:3000 --name ghayma-api-app ghayma-api
|
||||
```
|
||||
|
||||
### Test Containerized API
|
||||
```bash
|
||||
curl http://localhost:3000/api/v1/items
|
||||
```
|
||||
|
||||
### Stop Container
|
||||
```bash
|
||||
docker stop ghayma-api-app
|
||||
docker rm ghayma-api-app
|
||||
```
|
||||
@@ -0,0 +1,289 @@
|
||||
# Mortakaz Product Discovery & Cloud Integration Proposals
|
||||
|
||||
**Source Platform:** [mortakaz.com](https://mortakaz.com) (مرتكز — Arabic Products Platform)
|
||||
**Assessment:** Ghayma Cloud Integration Analysis
|
||||
**Date:** July 26, 2026
|
||||
|
||||
---
|
||||
|
||||
## Products Explored on Mortakaz
|
||||
|
||||
After browsing mortakaz.com, the following products were examined:
|
||||
|
||||
| # | Product | Category | Cloud Potential |
|
||||
| :--- | :--- | :--- | :--- |
|
||||
| 1 | **Taylor CV** (تايلور سي في) | Employment & AI | ⭐⭐⭐⭐⭐ |
|
||||
| 2 | **Athar** (أثر) | Islamic App | ⭐⭐ |
|
||||
| 3 | **Code City** (كود سيتي) | Developer Tools / 3D Visualization | ⭐⭐⭐⭐ |
|
||||
| 4 | **Social Browser** (سوشيال براوزر) | Web Browsers | ⭐⭐ |
|
||||
| 5 | **Workflow** (وورك فلو) | Enterprise SaaS | ⭐⭐⭐⭐⭐ |
|
||||
|
||||
### Selected Products for Integration Proposals
|
||||
|
||||
**Product A:** 🏆 **Workflow (وورك فلو)** — Enterprise SaaS Platform
|
||||
**Product B:** 🥈 **Taylor CV (تايلور سي في)** — AI-Powered Career Assistant
|
||||
|
||||
These two were selected because they are **cloud-native SaaS applications** with high computational demands, data persistence needs, and clear scalability requirements — making them ideal candidates for cloud platform integration.
|
||||
|
||||
---
|
||||
|
||||
---
|
||||
|
||||
# Integration Proposal 1: Workflow (وورك فلو)
|
||||
|
||||
## Product Description
|
||||
|
||||
**Workflow** is a centralized, multi-tenant SaaS platform designed to unify company management, project tracking, and financial operations into a single dashboard. It targets small-to-medium enterprises (SMEs) and holding companies with multiple subsidiaries.
|
||||
|
||||
**Key Features:**
|
||||
- Task & project management with Kanban boards
|
||||
- Invoicing, payroll, and financial reporting
|
||||
- Multi-tenant architecture (each company is isolated)
|
||||
- Employee management and HR operations
|
||||
- Real-time collaboration across teams and departments
|
||||
|
||||
**Source:** [mortakaz.com/projects/6a2eed27ec1d16fce885476a](https://www.mortakaz.com/projects/6a2eed27ec1d16fce885476a)
|
||||
|
||||
---
|
||||
|
||||
## Integration with Ghayma Cloud (ghaymah.systems)
|
||||
|
||||
### How to Integrate
|
||||
|
||||
Workflow can be deployed and scaled on Ghayma Cloud's container platform with persistent Block Storage for its multi-tenant data layer. The integration leverages three core Ghayma services:
|
||||
|
||||
1. **Ghayma Container Platform** — Host Workflow's backend API and frontend as containerized microservices
|
||||
2. **Ghayma Block Storage** — Persistent SSD volumes for PostgreSQL database (tenant data, invoices, payroll)
|
||||
3. **Ghayma Container Registry** — Store and version Docker images via CI/CD pipeline
|
||||
|
||||
### Architecture Sketch
|
||||
|
||||
```
|
||||
┌────────────────────────────────────────────────────────────────────────┐
|
||||
│ GHAYMA CLOUD PLATFORM │
|
||||
│ │
|
||||
│ ┌─────────────────────────────────────────────────────────────────┐ │
|
||||
│ │ NETWORKING LAYER │ │
|
||||
│ │ │ │
|
||||
│ │ Users ──▶ [DNS] ──▶ [CDN] ──▶ [Load Balancer (L7)] │ │
|
||||
│ │ │ │ │
|
||||
│ └────────────────────────────────────────┼────────────────────────┘ │
|
||||
│ │ │
|
||||
│ ┌────────────────────────────────────────┼────────────────────────┐ │
|
||||
│ │ COMPUTE LAYER │ │
|
||||
│ │ │ │
|
||||
│ │ ┌───────────────┐ ┌───────────────┐ ┌──────────────────┐ │ │
|
||||
│ │ │ Frontend │ │ Backend API │ │ Worker Service │ │ │
|
||||
│ │ │ (Vue/Nuxt) │ │ (Node.js / │ │ (Background │ │ │
|
||||
│ │ │ 3 replicas │ │ Python) │ │ jobs: payroll, │ │ │
|
||||
│ │ │ │ │ 5 replicas │ │ reports, email)│ │ │
|
||||
│ │ └───────────────┘ └───────┬───────┘ └────────┬─────────┘ │ │
|
||||
│ │ │ │ │ │
|
||||
│ └──────────────────────────────┼────────────────────┼─────────────┘ │
|
||||
│ │ │ │
|
||||
│ ┌──────────────────────────────┼────────────────────┼─────────────┐ │
|
||||
│ │ DATA LAYER │ │
|
||||
│ │ │ │
|
||||
│ │ ┌─────────────────────┐ ┌──────────────┐ ┌──────────────┐ │ │
|
||||
│ │ │ PostgreSQL DB │ │ Redis │ │ Ghayma │ │ │
|
||||
│ │ │ (Multi-tenant) │ │ (Cache + │ │ Block │ │ │
|
||||
│ │ │ │ │ Sessions) │ │ Storage │ │ │
|
||||
│ │ │ ┌───────────────┐ │ │ │ │ ┌─────────┐ │ │ │
|
||||
│ │ │ │ Tenant A Data │ │ └──────────────┘ │ │ DB Vol │ │ │ │
|
||||
│ │ │ │ Tenant B Data │ │ │ │ 200GB │ │ │ │
|
||||
│ │ │ │ Tenant C Data │ │ │ │ SSD │ │ │ │
|
||||
│ │ │ └───────────────┘ │ │ └─────────┘ │ │ │
|
||||
│ │ └──────────┬──────────┘ └──────────────┘ │ │
|
||||
│ │ │ │ │
|
||||
│ │ ▼ │ │
|
||||
│ │ Ghayma Block Storage Volume (persistent across restarts) │ │
|
||||
│ └─────────────────────────────────────────────────────────────────┘ │
|
||||
│ │
|
||||
│ ┌─────────────────────────────────────────────────────────────────┐ │
|
||||
│ │ CI/CD: GitHub Actions → Ghayma Registry │ │
|
||||
│ └─────────────────────────────────────────────────────────────────┘ │
|
||||
│ │
|
||||
└────────────────────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Value Added to End Users
|
||||
|
||||
| Value | Impact |
|
||||
| :--- | :--- |
|
||||
| **High Availability** | Multi-replica deployment ensures 99.9% uptime for all tenants |
|
||||
| **Data Sovereignty** | Hosting on Ghayma (regional Middle East cloud) keeps Arabic enterprise data within the region, complying with local regulations |
|
||||
| **Auto-Scaling** | Handle tenant growth automatically — from 10 to 10,000 companies without downtime |
|
||||
| **Persistent Data** | Block Storage guarantees financial records (invoices, payroll) survive any container crash or deployment |
|
||||
| **Faster Performance** | Regional cloud = lower latency for MENA-based SMEs vs. US/EU-hosted alternatives |
|
||||
| **Cost Efficiency** | Pay-per-use container pricing scales with actual tenant load instead of fixed server costs |
|
||||
|
||||
---
|
||||
|
||||
## Potential Challenges
|
||||
|
||||
| Challenge | Category | Mitigation |
|
||||
| :--- | :--- | :--- |
|
||||
| **Multi-tenant data isolation** | Technical | Use PostgreSQL schemas-per-tenant or row-level security policies |
|
||||
| **Database backup & DR** | Technical | Schedule Ghayma Block Storage snapshots every 6 hours |
|
||||
| **Compliance (financial data)** | Commercial | Ensure Ghayma Cloud meets local data residency regulations for financial records |
|
||||
| **Vendor lock-in** | Commercial | Keep application container-based (Docker) for portability to other cloud providers |
|
||||
| **Cold start latency** | Technical | Use pre-warmed container pool (min 2 replicas always running) |
|
||||
| **Integration complexity** | Technical | Gradual migration: start with backend API on Ghayma, keep DB local initially |
|
||||
|
||||
---
|
||||
|
||||
---
|
||||
|
||||
# Integration Proposal 2: Taylor CV (تايلور سي في)
|
||||
|
||||
## Product Description
|
||||
|
||||
**Taylor CV** is an AI-powered career assistant that helps Arabic-speaking job seekers land their target roles. It uses the user's knowledge base (skills, work history, projects) combined with a specific job description to generate optimized, ATS-friendly resumes and cover letters.
|
||||
|
||||
**Key Features:**
|
||||
- AI-generated customized resumes and cover letters
|
||||
- Interview preparation (Q&A + mock voice calls)
|
||||
- Daily learning plans to bridge skill gaps
|
||||
- Application tracking board
|
||||
- Multi-language support: Arabic, English, French
|
||||
- Bring-your-own API key option for free access
|
||||
|
||||
**Source:** [mortakaz.com/projects/6a60ad7e3d576288c1011ff2](https://www.mortakaz.com/projects/6a60ad7e3d576288c1011ff2)
|
||||
|
||||
---
|
||||
|
||||
## Integration with Ghayma Cloud & mithal.space
|
||||
|
||||
### How to Integrate
|
||||
|
||||
Taylor CV can be deployed on Ghayma Cloud with its AI processing pipeline leveraging containerized microservices. The integration with **mithal.space** (Ghayma's example/demo engine) can provide a public showcase and monitoring dashboard for the service.
|
||||
|
||||
1. **Ghayma Container Platform** — Host the web app, API, and AI worker services
|
||||
2. **Ghayma Block Storage** — Store user profiles, generated documents (PDFs), and conversation history
|
||||
3. **mithal.space Integration** — Publish Taylor CV as a featured example on mithal.space with live uptime monitoring, demonstrating Ghayma's AI workload capabilities
|
||||
|
||||
### Architecture Sketch
|
||||
|
||||
```
|
||||
┌────────────────────────────────────────────────────────────────────────┐
|
||||
│ GHAYMA CLOUD PLATFORM │
|
||||
│ │
|
||||
│ Users (Job Seekers) │
|
||||
│ │ │
|
||||
│ ▼ │
|
||||
│ ┌──────────────────────────────────────────────────────────────┐ │
|
||||
│ │ FRONTEND LAYER │ │
|
||||
│ │ Taylor CV Web App (Vue/React) │ │
|
||||
│ │ Hosted on Ghayma CDN + Container │ │
|
||||
│ └──────────────────────┬───────────────────────────────────────┘ │
|
||||
│ │ │
|
||||
│ ┌──────────────────────┼───────────────────────────────────────┐ │
|
||||
│ │ API GATEWAY │ │
|
||||
│ │ │ │ │
|
||||
│ │ ┌────────────┼──────────────┐ │ │
|
||||
│ │ │ │ │ │ │
|
||||
│ │ ▼ ▼ ▼ │ │
|
||||
│ │ ┌──────────┐ ┌──────────┐ ┌──────────────┐ │ │
|
||||
│ │ │ Auth │ │ Resume │ │ Interview │ │ │
|
||||
│ │ │ Service │ │ Builder │ │ Prep Service│ │ │
|
||||
│ │ │ │ │ Service │ │ (Voice AI) │ │ │
|
||||
│ │ └──────────┘ └────┬─────┘ └──────┬───────┘ │ │
|
||||
│ │ │ │ │ │
|
||||
│ │ ▼ ▼ │ │
|
||||
│ │ ┌─────────────────────────┐ │ │
|
||||
│ │ │ AI Processing Queue │ │ │
|
||||
│ │ │ (Background Workers) │ │ │
|
||||
│ │ │ • PDF Generation │ │ │
|
||||
│ │ │ • LLM API Calls │ │ │
|
||||
│ │ │ • Voice Transcription │ │ │
|
||||
│ │ └────────────┬────────────┘ │ │
|
||||
│ │ │ │ │
|
||||
│ └───────────────────────────┼───────────────────────────────────┘ │
|
||||
│ │ │
|
||||
│ ┌───────────────────────────┼───────────────────────────────────┐ │
|
||||
│ │ DATA LAYER │ │
|
||||
│ │ │ │
|
||||
│ │ ┌──────────────┐ ┌──────────┐ ┌───────────────────────┐ │ │
|
||||
│ │ │ MongoDB / │ │ Redis │ │ Ghayma Block Storage │ │ │
|
||||
│ │ │ PostgreSQL │ │ (Cache │ │ • User profiles │ │ │
|
||||
│ │ │ (User data, │ │ + Job │ │ • Generated PDFs │ │ │
|
||||
│ │ │ resumes) │ │ Queue) │ │ • Voice recordings │ │ │
|
||||
│ │ └──────────────┘ └──────────┘ └───────────────────────┘ │ │
|
||||
│ └───────────────────────────────────────────────────────────────┘ │
|
||||
│ │
|
||||
│ ┌───────────────────────────────────────────────────────────────┐ │
|
||||
│ │ mithal.space SHOWCASE INTEGRATION │ │
|
||||
│ │ │ │
|
||||
│ │ • Live demo page on mithal.space featuring Taylor CV │ │
|
||||
│ │ • Uptime monitoring dashboard (99.9% SLA display) │ │
|
||||
│ │ • Public API status page for job seekers │ │
|
||||
│ │ • Case study: "AI Workloads on Ghayma Cloud" │ │
|
||||
│ └───────────────────────────────────────────────────────────────┘ │
|
||||
│ │
|
||||
└────────────────────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Value Added to End Users
|
||||
|
||||
| Value | Impact |
|
||||
| :--- | :--- |
|
||||
| **Low Latency AI** | Running AI workers on Ghayma's regional cloud reduces round-trip time for Arabic NLP processing |
|
||||
| **Scalable AI Workers** | Auto-scale GPU/CPU workers during peak job application seasons (graduation, Ramadan hiring) |
|
||||
| **Data Privacy** | Users' personal career data (skills, work history) stays in-region, not on US servers |
|
||||
| **Reliable PDF Generation** | Block Storage ensures generated resumes are never lost, even during worker crashes |
|
||||
| **mithal.space Showcase** | Public visibility as a featured Ghayma Cloud success story drives user adoption |
|
||||
| **Cost-Effective AI** | Container-based workers spin up only when processing requests, reducing idle compute costs |
|
||||
|
||||
---
|
||||
|
||||
## Potential Challenges
|
||||
|
||||
| Challenge | Category | Mitigation |
|
||||
| :--- | :--- | :--- |
|
||||
| **LLM API costs** | Commercial | Implement usage tiers and the "bring your own API key" model to offload costs |
|
||||
| **Arabic NLP accuracy** | Technical | Fine-tune models on Arabic resume datasets; leverage community feedback from mortakaz |
|
||||
| **GPU availability** | Technical | Pre-reserve GPU instances on Ghayma Cloud; fall back to CPU for smaller workloads |
|
||||
| **Voice AI latency** | Technical | Deploy WebSocket-based streaming for real-time mock interviews |
|
||||
| **User data sensitivity** | Commercial | Implement end-to-end encryption for stored resumes and personal data |
|
||||
| **Peak load spikes** | Technical | Pre-scale workers during known peak periods (September, January hiring seasons) |
|
||||
|
||||
---
|
||||
|
||||
---
|
||||
|
||||
# Final Recommendation: Most Viable Product
|
||||
|
||||
## 🏆 Winner: Workflow (وورك فلو)
|
||||
|
||||
### Why Workflow is the Most Viable Integration
|
||||
|
||||
| Criteria | Workflow | Taylor CV |
|
||||
| :--- | :--- | :--- |
|
||||
| **Revenue Model** | ✅ SaaS subscriptions (recurring, predictable) | ⚠️ Freemium with API-key model (uncertain revenue) |
|
||||
| **Cloud Dependency** | ✅ Critical — needs persistent DB, multi-tenancy, scaling | ⚠️ Moderate — AI can run on any cloud |
|
||||
| **Data Persistence Need** | ✅ Must-have — financial records, invoices, payroll | ⚠️ Nice-to-have — resumes can be re-generated |
|
||||
| **Target Market Size** | ✅ Large — every SME in MENA region (millions) | ⚠️ Niche — job seekers (cyclical demand) |
|
||||
| **Regulatory Fit** | ✅ Financial data residency laws favor regional cloud | ⚠️ Less regulatory pressure |
|
||||
| **Long-term Stickiness** | ✅ High — companies rarely migrate ERP/management tools | ⚠️ Low — users may switch AI tools easily |
|
||||
| **Ghayma Revenue Potential** | ✅ High — persistent storage + compute + DB | ⚠️ Moderate — burst compute only |
|
||||
|
||||
### Summary
|
||||
|
||||
**Workflow** is the most viable because:
|
||||
|
||||
1. **It needs the cloud** — A multi-tenant SaaS managing invoices, payroll, and employee data **requires** reliable cloud infrastructure with persistent storage. This isn't optional; it's fundamental.
|
||||
|
||||
2. **Recurring revenue for both parties** — Workflow charges monthly SaaS fees, and Ghayma earns from persistent compute + storage. Both revenue streams are predictable.
|
||||
|
||||
3. **Data residency is a competitive advantage** — MENA enterprises increasingly require financial data to stay in-region. Hosting on Ghayma Cloud (regional) is a selling point competitors on AWS US-East can't match.
|
||||
|
||||
4. **High stickiness** — Once a company moves its invoicing, HR, and project management to Workflow on Ghayma, switching costs are extremely high, creating a long-term customer for both Workflow and Ghayma Cloud.
|
||||
|
||||
---
|
||||
|
||||
> **Report prepared for:** Ghayma Assessment — Product Discovery & Integration section
|
||||
> **Last updated:** July 26, 2026
|
||||
@@ -0,0 +1,12 @@
|
||||
FROM node:20-alpine
|
||||
|
||||
WORKDIR /app
|
||||
|
||||
COPY package*.json ./
|
||||
RUN npm ci --only=production
|
||||
|
||||
COPY . .
|
||||
|
||||
EXPOSE 3000
|
||||
|
||||
CMD ["npm","start"]
|
||||
@@ -0,0 +1,148 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
Monitoring Script for Ghayma REST API
|
||||
Scans the application health endpoint every 30 seconds and logs results.
|
||||
"""
|
||||
|
||||
import urllib.request
|
||||
import urllib.error
|
||||
import json
|
||||
import time
|
||||
import datetime
|
||||
import os
|
||||
import sys
|
||||
|
||||
# Configuration
|
||||
APP_URL = os.environ.get("APP_URL", "http://localhost:3000")
|
||||
HEALTH_ENDPOINT = f"{APP_URL}/health"
|
||||
CHECK_INTERVAL = 30 # seconds
|
||||
LOG_FILE = os.path.join(os.path.dirname(os.path.abspath(__file__)), "monitor.log")
|
||||
TIMEOUT = 5 # seconds
|
||||
|
||||
# ANSI colors for terminal output
|
||||
GREEN = "\033[92m"
|
||||
RED = "\033[91m"
|
||||
YELLOW = "\033[93m"
|
||||
CYAN = "\033[96m"
|
||||
RESET = "\033[0m"
|
||||
BOLD = "\033[1m"
|
||||
|
||||
|
||||
def log(message, level="INFO"):
|
||||
"""Log a message to both console and log file."""
|
||||
timestamp = datetime.datetime.now().strftime("%Y-%m-%d %H:%M:%S")
|
||||
log_entry = f"[{timestamp}] [{level}] {message}"
|
||||
|
||||
# Color-coded console output
|
||||
if level == "OK":
|
||||
print(f"{GREEN}{log_entry}{RESET}")
|
||||
elif level == "ERROR":
|
||||
print(f"{RED}{BOLD}{log_entry}{RESET}")
|
||||
elif level == "WARN":
|
||||
print(f"{YELLOW}{log_entry}{RESET}")
|
||||
else:
|
||||
print(f"{CYAN}{log_entry}{RESET}")
|
||||
|
||||
# Append to log file
|
||||
with open(LOG_FILE, "a", encoding="utf-8") as f:
|
||||
f.write(log_entry + "\n")
|
||||
|
||||
|
||||
def check_health():
|
||||
"""Send a GET request to the health endpoint and return the result."""
|
||||
try:
|
||||
req = urllib.request.Request(HEALTH_ENDPOINT, method="GET")
|
||||
start_time = time.time()
|
||||
response = urllib.request.urlopen(req, timeout=TIMEOUT)
|
||||
response_time_ms = round((time.time() - start_time) * 1000)
|
||||
status_code = response.status
|
||||
body = json.loads(response.read().decode("utf-8"))
|
||||
|
||||
return {
|
||||
"healthy": status_code == 200 and body.get("status") == "UP",
|
||||
"status_code": status_code,
|
||||
"response_time_ms": response_time_ms,
|
||||
"uptime": body.get("uptime"),
|
||||
"error": None,
|
||||
}
|
||||
|
||||
except urllib.error.HTTPError as e:
|
||||
return {
|
||||
"healthy": False,
|
||||
"status_code": e.code,
|
||||
"response_time_ms": None,
|
||||
"uptime": None,
|
||||
"error": f"HTTP {e.code}: {e.reason}",
|
||||
}
|
||||
|
||||
except urllib.error.URLError as e:
|
||||
return {
|
||||
"healthy": False,
|
||||
"status_code": None,
|
||||
"response_time_ms": None,
|
||||
"uptime": None,
|
||||
"error": f"Connection failed: {e.reason}",
|
||||
}
|
||||
|
||||
except Exception as e:
|
||||
return {
|
||||
"healthy": False,
|
||||
"status_code": None,
|
||||
"response_time_ms": None,
|
||||
"uptime": None,
|
||||
"error": str(e),
|
||||
}
|
||||
|
||||
|
||||
def main():
|
||||
consecutive_failures = 0
|
||||
|
||||
print(f"{BOLD}{CYAN}")
|
||||
print("=" * 55)
|
||||
print(" Ghayma REST API - Health Monitor")
|
||||
print(f" Target: {HEALTH_ENDPOINT}")
|
||||
print(f" Interval: {CHECK_INTERVAL}s")
|
||||
print(f" Log file: {LOG_FILE}")
|
||||
print("=" * 55)
|
||||
print(f"{RESET}")
|
||||
|
||||
log(f"Monitor started | Target: {HEALTH_ENDPOINT} | Interval: {CHECK_INTERVAL}s")
|
||||
|
||||
try:
|
||||
while True:
|
||||
result = check_health()
|
||||
|
||||
if result["healthy"]:
|
||||
consecutive_failures = 0
|
||||
uptime_str = f"{result['uptime']:.1f}s" if result["uptime"] else "N/A"
|
||||
log(
|
||||
f"HEALTHY | Status: {result['status_code']} | "
|
||||
f"Response: {result['response_time_ms']}ms | "
|
||||
f"Uptime: {uptime_str}",
|
||||
level="OK",
|
||||
)
|
||||
else:
|
||||
consecutive_failures += 1
|
||||
log(
|
||||
f"UNHEALTHY | Error: {result['error']} | "
|
||||
f"Consecutive failures: {consecutive_failures}",
|
||||
level="ERROR",
|
||||
)
|
||||
|
||||
if consecutive_failures >= 3:
|
||||
log(
|
||||
f"ALERT: {consecutive_failures} consecutive failures! "
|
||||
f"Application may be DOWN.",
|
||||
level="WARN",
|
||||
)
|
||||
|
||||
time.sleep(CHECK_INTERVAL)
|
||||
|
||||
except KeyboardInterrupt:
|
||||
log("Monitor stopped by user.", level="INFO")
|
||||
print(f"\n{YELLOW}Monitor stopped.{RESET}")
|
||||
sys.exit(0)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,764 @@
|
||||
<!DOCTYPE html>
|
||||
<html lang="en">
|
||||
<head>
|
||||
<meta charset="UTF-8">
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1.0">
|
||||
<title>Ghayma API — Monitoring Dashboard</title>
|
||||
<meta name="description" content="Real-time monitoring dashboard for Ghayma REST API">
|
||||
<link href="https://fonts.googleapis.com/css2?family=Inter:wght@300;400;500;600;700;800&display=swap" rel="stylesheet">
|
||||
<style>
|
||||
*, *::before, *::after { margin: 0; padding: 0; box-sizing: border-box; }
|
||||
|
||||
:root {
|
||||
--bg-primary: #0a0e1a;
|
||||
--bg-secondary: #111827;
|
||||
--bg-card: rgba(17, 24, 39, 0.7);
|
||||
--border-subtle: rgba(255, 255, 255, 0.06);
|
||||
--text-primary: #f1f5f9;
|
||||
--text-secondary: #94a3b8;
|
||||
--text-muted: #64748b;
|
||||
--accent-green: #22c55e;
|
||||
--accent-green-glow: rgba(34, 197, 94, 0.15);
|
||||
--accent-red: #ef4444;
|
||||
--accent-red-glow: rgba(239, 68, 68, 0.15);
|
||||
--accent-blue: #3b82f6;
|
||||
--accent-blue-glow: rgba(59, 130, 246, 0.12);
|
||||
--accent-amber: #f59e0b;
|
||||
--accent-purple: #a855f7;
|
||||
--accent-purple-glow: rgba(168, 85, 247, 0.12);
|
||||
--radius: 16px;
|
||||
--radius-sm: 10px;
|
||||
}
|
||||
|
||||
body {
|
||||
font-family: 'Inter', -apple-system, sans-serif;
|
||||
background: var(--bg-primary);
|
||||
color: var(--text-primary);
|
||||
min-height: 100vh;
|
||||
overflow-x: hidden;
|
||||
}
|
||||
|
||||
/* Ambient background effects */
|
||||
body::before {
|
||||
content: '';
|
||||
position: fixed;
|
||||
top: -50%;
|
||||
left: -50%;
|
||||
width: 200%;
|
||||
height: 200%;
|
||||
background: radial-gradient(ellipse at 20% 50%, rgba(59, 130, 246, 0.04) 0%, transparent 50%),
|
||||
radial-gradient(ellipse at 80% 20%, rgba(168, 85, 247, 0.03) 0%, transparent 50%),
|
||||
radial-gradient(ellipse at 50% 80%, rgba(34, 197, 94, 0.03) 0%, transparent 50%);
|
||||
z-index: 0;
|
||||
pointer-events: none;
|
||||
}
|
||||
|
||||
.app { position: relative; z-index: 1; }
|
||||
|
||||
/* Header */
|
||||
.header {
|
||||
padding: 28px 40px;
|
||||
display: flex;
|
||||
align-items: center;
|
||||
justify-content: space-between;
|
||||
border-bottom: 1px solid var(--border-subtle);
|
||||
backdrop-filter: blur(20px);
|
||||
background: rgba(10, 14, 26, 0.6);
|
||||
position: sticky;
|
||||
top: 0;
|
||||
z-index: 100;
|
||||
}
|
||||
|
||||
.header-left { display: flex; align-items: center; gap: 14px; }
|
||||
|
||||
.logo {
|
||||
width: 40px;
|
||||
height: 40px;
|
||||
border-radius: 12px;
|
||||
background: linear-gradient(135deg, var(--accent-blue), var(--accent-purple));
|
||||
display: flex;
|
||||
align-items: center;
|
||||
justify-content: center;
|
||||
font-size: 18px;
|
||||
font-weight: 800;
|
||||
color: #fff;
|
||||
box-shadow: 0 4px 20px rgba(59, 130, 246, 0.25);
|
||||
}
|
||||
|
||||
.header-title {
|
||||
font-size: 20px;
|
||||
font-weight: 700;
|
||||
letter-spacing: -0.3px;
|
||||
}
|
||||
|
||||
.header-title span { color: var(--text-muted); font-weight: 400; }
|
||||
|
||||
.header-right { display: flex; align-items: center; gap: 16px; }
|
||||
|
||||
.status-badge {
|
||||
display: flex;
|
||||
align-items: center;
|
||||
gap: 8px;
|
||||
padding: 8px 16px;
|
||||
border-radius: 100px;
|
||||
font-size: 13px;
|
||||
font-weight: 600;
|
||||
letter-spacing: 0.3px;
|
||||
transition: all 0.4s ease;
|
||||
}
|
||||
|
||||
.status-badge.online {
|
||||
background: var(--accent-green-glow);
|
||||
color: var(--accent-green);
|
||||
border: 1px solid rgba(34, 197, 94, 0.2);
|
||||
}
|
||||
|
||||
.status-badge.offline {
|
||||
background: var(--accent-red-glow);
|
||||
color: var(--accent-red);
|
||||
border: 1px solid rgba(239, 68, 68, 0.2);
|
||||
}
|
||||
|
||||
.status-dot {
|
||||
width: 8px;
|
||||
height: 8px;
|
||||
border-radius: 50%;
|
||||
position: relative;
|
||||
}
|
||||
|
||||
.status-badge.online .status-dot { background: var(--accent-green); }
|
||||
.status-badge.offline .status-dot { background: var(--accent-red); }
|
||||
|
||||
.status-dot::after {
|
||||
content: '';
|
||||
position: absolute;
|
||||
top: -3px;
|
||||
left: -3px;
|
||||
width: 14px;
|
||||
height: 14px;
|
||||
border-radius: 50%;
|
||||
animation: pulse-ring 2s ease-out infinite;
|
||||
}
|
||||
|
||||
.status-badge.online .status-dot::after { background: rgba(34, 197, 94, 0.3); }
|
||||
.status-badge.offline .status-dot::after { background: rgba(239, 68, 68, 0.3); }
|
||||
|
||||
@keyframes pulse-ring {
|
||||
0% { transform: scale(1); opacity: 1; }
|
||||
100% { transform: scale(2.2); opacity: 0; }
|
||||
}
|
||||
|
||||
.last-updated {
|
||||
font-size: 12px;
|
||||
color: var(--text-muted);
|
||||
}
|
||||
|
||||
/* Main content */
|
||||
.main { padding: 32px 40px; max-width: 1400px; margin: 0 auto; }
|
||||
|
||||
/* Cards grid */
|
||||
.metrics-grid {
|
||||
display: grid;
|
||||
grid-template-columns: repeat(4, 1fr);
|
||||
gap: 20px;
|
||||
margin-bottom: 32px;
|
||||
}
|
||||
|
||||
.metric-card {
|
||||
background: var(--bg-card);
|
||||
border: 1px solid var(--border-subtle);
|
||||
border-radius: var(--radius);
|
||||
padding: 24px;
|
||||
backdrop-filter: blur(12px);
|
||||
transition: all 0.3s ease;
|
||||
position: relative;
|
||||
overflow: hidden;
|
||||
}
|
||||
|
||||
.metric-card::before {
|
||||
content: '';
|
||||
position: absolute;
|
||||
top: 0;
|
||||
left: 0;
|
||||
right: 0;
|
||||
height: 2px;
|
||||
border-radius: var(--radius) var(--radius) 0 0;
|
||||
opacity: 0;
|
||||
transition: opacity 0.3s ease;
|
||||
}
|
||||
|
||||
.metric-card:hover { border-color: rgba(255, 255, 255, 0.1); transform: translateY(-2px); }
|
||||
.metric-card:hover::before { opacity: 1; }
|
||||
|
||||
.metric-card.blue::before { background: linear-gradient(90deg, var(--accent-blue), transparent); }
|
||||
.metric-card.green::before { background: linear-gradient(90deg, var(--accent-green), transparent); }
|
||||
.metric-card.amber::before { background: linear-gradient(90deg, var(--accent-amber), transparent); }
|
||||
.metric-card.purple::before { background: linear-gradient(90deg, var(--accent-purple), transparent); }
|
||||
|
||||
.metric-icon {
|
||||
width: 42px;
|
||||
height: 42px;
|
||||
border-radius: 12px;
|
||||
display: flex;
|
||||
align-items: center;
|
||||
justify-content: center;
|
||||
font-size: 20px;
|
||||
margin-bottom: 18px;
|
||||
}
|
||||
|
||||
.metric-card.blue .metric-icon { background: var(--accent-blue-glow); }
|
||||
.metric-card.green .metric-icon { background: var(--accent-green-glow); }
|
||||
.metric-card.amber .metric-icon { background: rgba(245, 158, 11, 0.12); }
|
||||
.metric-card.purple .metric-icon { background: var(--accent-purple-glow); }
|
||||
|
||||
.metric-label {
|
||||
font-size: 13px;
|
||||
color: var(--text-muted);
|
||||
font-weight: 500;
|
||||
margin-bottom: 6px;
|
||||
text-transform: uppercase;
|
||||
letter-spacing: 0.6px;
|
||||
}
|
||||
|
||||
.metric-value {
|
||||
font-size: 32px;
|
||||
font-weight: 800;
|
||||
letter-spacing: -1px;
|
||||
line-height: 1;
|
||||
}
|
||||
|
||||
.metric-card.blue .metric-value { color: var(--accent-blue); }
|
||||
.metric-card.green .metric-value { color: var(--accent-green); }
|
||||
.metric-card.amber .metric-value { color: var(--accent-amber); }
|
||||
.metric-card.purple .metric-value { color: var(--accent-purple); }
|
||||
|
||||
.metric-sub {
|
||||
font-size: 12px;
|
||||
color: var(--text-muted);
|
||||
margin-top: 8px;
|
||||
}
|
||||
|
||||
/* Panels grid */
|
||||
.panels-grid {
|
||||
display: grid;
|
||||
grid-template-columns: 2fr 1fr;
|
||||
gap: 20px;
|
||||
margin-bottom: 32px;
|
||||
}
|
||||
|
||||
.panel {
|
||||
background: var(--bg-card);
|
||||
border: 1px solid var(--border-subtle);
|
||||
border-radius: var(--radius);
|
||||
backdrop-filter: blur(12px);
|
||||
overflow: hidden;
|
||||
}
|
||||
|
||||
.panel-header {
|
||||
padding: 20px 24px;
|
||||
border-bottom: 1px solid var(--border-subtle);
|
||||
display: flex;
|
||||
align-items: center;
|
||||
justify-content: space-between;
|
||||
}
|
||||
|
||||
.panel-title {
|
||||
font-size: 15px;
|
||||
font-weight: 600;
|
||||
display: flex;
|
||||
align-items: center;
|
||||
gap: 8px;
|
||||
}
|
||||
|
||||
.panel-body { padding: 24px; }
|
||||
|
||||
/* Response time chart */
|
||||
.chart-container {
|
||||
height: 200px;
|
||||
display: flex;
|
||||
align-items: flex-end;
|
||||
gap: 6px;
|
||||
padding-top: 10px;
|
||||
}
|
||||
|
||||
.chart-bar-wrapper {
|
||||
flex: 1;
|
||||
display: flex;
|
||||
flex-direction: column;
|
||||
align-items: center;
|
||||
gap: 6px;
|
||||
height: 100%;
|
||||
justify-content: flex-end;
|
||||
}
|
||||
|
||||
.chart-bar {
|
||||
width: 100%;
|
||||
min-height: 4px;
|
||||
border-radius: 4px 4px 2px 2px;
|
||||
background: linear-gradient(to top, var(--accent-blue), rgba(59, 130, 246, 0.6));
|
||||
transition: height 0.5s cubic-bezier(0.22, 1, 0.36, 1);
|
||||
position: relative;
|
||||
}
|
||||
|
||||
.chart-bar:hover {
|
||||
background: linear-gradient(to top, var(--accent-purple), rgba(168, 85, 247, 0.7));
|
||||
}
|
||||
|
||||
.chart-bar-label {
|
||||
font-size: 10px;
|
||||
color: var(--text-muted);
|
||||
writing-mode: horizontal-tb;
|
||||
}
|
||||
|
||||
.chart-empty {
|
||||
height: 100%;
|
||||
display: flex;
|
||||
align-items: center;
|
||||
justify-content: center;
|
||||
color: var(--text-muted);
|
||||
font-size: 13px;
|
||||
}
|
||||
|
||||
/* Endpoint list */
|
||||
.endpoint-list { list-style: none; }
|
||||
|
||||
.endpoint-item {
|
||||
display: flex;
|
||||
align-items: center;
|
||||
justify-content: space-between;
|
||||
padding: 12px 0;
|
||||
border-bottom: 1px solid var(--border-subtle);
|
||||
transition: all 0.2s ease;
|
||||
}
|
||||
|
||||
.endpoint-item:last-child { border-bottom: none; }
|
||||
.endpoint-item:hover { padding-left: 6px; }
|
||||
|
||||
.endpoint-name {
|
||||
font-size: 13px;
|
||||
font-weight: 500;
|
||||
font-family: 'Inter', monospace;
|
||||
color: var(--text-primary);
|
||||
}
|
||||
|
||||
.endpoint-count {
|
||||
font-size: 13px;
|
||||
font-weight: 700;
|
||||
color: var(--accent-blue);
|
||||
background: var(--accent-blue-glow);
|
||||
padding: 3px 10px;
|
||||
border-radius: 100px;
|
||||
}
|
||||
|
||||
.endpoint-empty {
|
||||
padding: 20px 0;
|
||||
text-align: center;
|
||||
color: var(--text-muted);
|
||||
font-size: 13px;
|
||||
}
|
||||
|
||||
/* Status codes panel */
|
||||
.status-codes-grid {
|
||||
display: grid;
|
||||
grid-template-columns: repeat(2, 1fr);
|
||||
gap: 12px;
|
||||
}
|
||||
|
||||
.status-code-card {
|
||||
background: rgba(255, 255, 255, 0.03);
|
||||
border: 1px solid var(--border-subtle);
|
||||
border-radius: var(--radius-sm);
|
||||
padding: 16px;
|
||||
text-align: center;
|
||||
transition: all 0.3s ease;
|
||||
}
|
||||
|
||||
.status-code-card:hover { background: rgba(255, 255, 255, 0.05); }
|
||||
|
||||
.status-code-value {
|
||||
font-size: 24px;
|
||||
font-weight: 800;
|
||||
margin-bottom: 4px;
|
||||
}
|
||||
|
||||
.status-code-label {
|
||||
font-size: 11px;
|
||||
color: var(--text-muted);
|
||||
text-transform: uppercase;
|
||||
letter-spacing: 0.5px;
|
||||
font-weight: 500;
|
||||
}
|
||||
|
||||
.code-2xx .status-code-value { color: var(--accent-green); }
|
||||
.code-3xx .status-code-value { color: var(--accent-blue); }
|
||||
.code-4xx .status-code-value { color: var(--accent-amber); }
|
||||
.code-5xx .status-code-value { color: var(--accent-red); }
|
||||
|
||||
/* Bottom panels */
|
||||
.bottom-grid {
|
||||
display: grid;
|
||||
grid-template-columns: 1fr 1fr;
|
||||
gap: 20px;
|
||||
}
|
||||
|
||||
/* Uptime display */
|
||||
.uptime-display {
|
||||
display: flex;
|
||||
gap: 16px;
|
||||
flex-wrap: wrap;
|
||||
}
|
||||
|
||||
.uptime-unit {
|
||||
flex: 1;
|
||||
min-width: 80px;
|
||||
text-align: center;
|
||||
background: rgba(255, 255, 255, 0.03);
|
||||
border: 1px solid var(--border-subtle);
|
||||
border-radius: var(--radius-sm);
|
||||
padding: 16px 12px;
|
||||
}
|
||||
|
||||
.uptime-value {
|
||||
font-size: 28px;
|
||||
font-weight: 800;
|
||||
color: var(--accent-blue);
|
||||
line-height: 1;
|
||||
}
|
||||
|
||||
.uptime-label {
|
||||
font-size: 11px;
|
||||
color: var(--text-muted);
|
||||
margin-top: 6px;
|
||||
text-transform: uppercase;
|
||||
letter-spacing: 0.5px;
|
||||
}
|
||||
|
||||
/* Request log */
|
||||
.request-log {
|
||||
max-height: 180px;
|
||||
overflow-y: auto;
|
||||
scrollbar-width: thin;
|
||||
scrollbar-color: rgba(255, 255, 255, 0.1) transparent;
|
||||
}
|
||||
|
||||
.log-entry {
|
||||
display: flex;
|
||||
align-items: center;
|
||||
gap: 10px;
|
||||
padding: 8px 0;
|
||||
font-size: 12px;
|
||||
border-bottom: 1px solid rgba(255, 255, 255, 0.03);
|
||||
animation: fadeSlideIn 0.3s ease;
|
||||
}
|
||||
|
||||
@keyframes fadeSlideIn {
|
||||
from { opacity: 0; transform: translateX(-8px); }
|
||||
to { opacity: 1; transform: translateX(0); }
|
||||
}
|
||||
|
||||
.log-time { color: var(--text-muted); font-family: monospace; white-space: nowrap; }
|
||||
.log-status {
|
||||
padding: 2px 8px;
|
||||
border-radius: 4px;
|
||||
font-weight: 600;
|
||||
font-size: 11px;
|
||||
}
|
||||
.log-status.ok { background: var(--accent-green-glow); color: var(--accent-green); }
|
||||
.log-status.err { background: var(--accent-red-glow); color: var(--accent-red); }
|
||||
.log-message { color: var(--text-secondary); flex: 1; }
|
||||
|
||||
/* Responsive */
|
||||
@media (max-width: 1024px) {
|
||||
.metrics-grid { grid-template-columns: repeat(2, 1fr); }
|
||||
.panels-grid, .bottom-grid { grid-template-columns: 1fr; }
|
||||
.main { padding: 20px; }
|
||||
.header { padding: 20px; }
|
||||
}
|
||||
|
||||
@media (max-width: 640px) {
|
||||
.metrics-grid { grid-template-columns: 1fr; }
|
||||
}
|
||||
</style>
|
||||
</head>
|
||||
<body>
|
||||
<div class="app">
|
||||
<!-- Header -->
|
||||
<header class="header">
|
||||
<div class="header-left">
|
||||
<div class="logo">G</div>
|
||||
<h1 class="header-title">Ghayma <span>Monitoring</span></h1>
|
||||
</div>
|
||||
<div class="header-right">
|
||||
<div id="statusBadge" class="status-badge offline">
|
||||
<div class="status-dot"></div>
|
||||
<span id="statusText">CONNECTING...</span>
|
||||
</div>
|
||||
<div class="last-updated">Updated: <span id="lastUpdated">—</span></div>
|
||||
</div>
|
||||
</header>
|
||||
|
||||
<!-- Main Content -->
|
||||
<main class="main">
|
||||
<!-- Metric Cards -->
|
||||
<section class="metrics-grid">
|
||||
<div class="metric-card blue">
|
||||
<div class="metric-icon">📊</div>
|
||||
<div class="metric-label">Total Requests</div>
|
||||
<div class="metric-value" id="totalRequests">0</div>
|
||||
<div class="metric-sub" id="requestRate">0 req/min</div>
|
||||
</div>
|
||||
<div class="metric-card green">
|
||||
<div class="metric-icon">✅</div>
|
||||
<div class="metric-label">Successful</div>
|
||||
<div class="metric-value" id="successRequests">0</div>
|
||||
<div class="metric-sub" id="successRate">0% success rate</div>
|
||||
</div>
|
||||
<div class="metric-card amber">
|
||||
<div class="metric-icon">⚡</div>
|
||||
<div class="metric-label">Avg Response Time</div>
|
||||
<div class="metric-value" id="avgResponseTime">0<span style="font-size:16px;font-weight:400;"> ms</span></div>
|
||||
<div class="metric-sub">Last 100 requests</div>
|
||||
</div>
|
||||
<div class="metric-card purple">
|
||||
<div class="metric-icon">❌</div>
|
||||
<div class="metric-label">Errors</div>
|
||||
<div class="metric-value" id="errorRequests">0</div>
|
||||
<div class="metric-sub" id="errorRate">0% error rate</div>
|
||||
</div>
|
||||
</section>
|
||||
|
||||
<!-- Response Time Chart + Endpoints -->
|
||||
<section class="panels-grid">
|
||||
<div class="panel">
|
||||
<div class="panel-header">
|
||||
<div class="panel-title">⏱️ Response Time (last 20 requests)</div>
|
||||
</div>
|
||||
<div class="panel-body">
|
||||
<div class="chart-container" id="responseChart">
|
||||
<div class="chart-empty">Waiting for data...</div>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<div class="panel">
|
||||
<div class="panel-header">
|
||||
<div class="panel-title">🔗 Requests by Endpoint</div>
|
||||
</div>
|
||||
<div class="panel-body">
|
||||
<ul class="endpoint-list" id="endpointList">
|
||||
<li class="endpoint-empty">No requests yet</li>
|
||||
</ul>
|
||||
</div>
|
||||
</div>
|
||||
</section>
|
||||
|
||||
<!-- Bottom: Uptime + Status Codes -->
|
||||
<section class="bottom-grid">
|
||||
<div class="panel">
|
||||
<div class="panel-header">
|
||||
<div class="panel-title">🕐 Uptime</div>
|
||||
</div>
|
||||
<div class="panel-body">
|
||||
<div class="uptime-display" id="uptimeDisplay">
|
||||
<div class="uptime-unit"><div class="uptime-value" id="uptDays">0</div><div class="uptime-label">Days</div></div>
|
||||
<div class="uptime-unit"><div class="uptime-value" id="uptHours">0</div><div class="uptime-label">Hours</div></div>
|
||||
<div class="uptime-unit"><div class="uptime-value" id="uptMins">0</div><div class="uptime-label">Minutes</div></div>
|
||||
<div class="uptime-unit"><div class="uptime-value" id="uptSecs">0</div><div class="uptime-label">Seconds</div></div>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<div class="panel">
|
||||
<div class="panel-header">
|
||||
<div class="panel-title">📋 Status Codes</div>
|
||||
</div>
|
||||
<div class="panel-body">
|
||||
<div class="status-codes-grid" id="statusCodesGrid">
|
||||
<div class="status-code-card code-2xx">
|
||||
<div class="status-code-value" id="code2xx">0</div>
|
||||
<div class="status-code-label">2xx Success</div>
|
||||
</div>
|
||||
<div class="status-code-card code-3xx">
|
||||
<div class="status-code-value" id="code3xx">0</div>
|
||||
<div class="status-code-label">3xx Redirect</div>
|
||||
</div>
|
||||
<div class="status-code-card code-4xx">
|
||||
<div class="status-code-value" id="code4xx">0</div>
|
||||
<div class="status-code-label">4xx Client Err</div>
|
||||
</div>
|
||||
<div class="status-code-card code-5xx">
|
||||
<div class="status-code-value" id="code5xx">0</div>
|
||||
<div class="status-code-label">5xx Server Err</div>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
</section>
|
||||
</main>
|
||||
</div>
|
||||
|
||||
<script>
|
||||
const API_BASE = window.location.origin;
|
||||
const POLL_INTERVAL = 5000; // 5 seconds
|
||||
let previousTotal = 0;
|
||||
let previousTime = Date.now();
|
||||
|
||||
async function fetchMetrics() {
|
||||
try {
|
||||
const response = await fetch(`${API_BASE}/metrics`);
|
||||
if (!response.ok) throw new Error(`HTTP ${response.status}`);
|
||||
const data = await response.json();
|
||||
updateDashboard(data);
|
||||
} catch (err) {
|
||||
setOffline(err.message);
|
||||
}
|
||||
}
|
||||
|
||||
function updateDashboard(data) {
|
||||
// Status badge
|
||||
const badge = document.getElementById('statusBadge');
|
||||
const statusText = document.getElementById('statusText');
|
||||
badge.className = 'status-badge online';
|
||||
statusText.textContent = 'ONLINE';
|
||||
|
||||
// Last updated
|
||||
document.getElementById('lastUpdated').textContent =
|
||||
new Date().toLocaleTimeString();
|
||||
|
||||
// Metric cards
|
||||
animateValue('totalRequests', data.totalRequests);
|
||||
animateValue('successRequests', data.successRequests);
|
||||
document.getElementById('avgResponseTime').innerHTML =
|
||||
`${data.avgResponseTime}<span style="font-size:16px;font-weight:400;"> ms</span>`;
|
||||
animateValue('errorRequests', data.errorRequests);
|
||||
|
||||
// Rates
|
||||
const now = Date.now();
|
||||
const elapsed = (now - previousTime) / 60000; // minutes
|
||||
if (elapsed > 0 && previousTotal > 0) {
|
||||
const rate = ((data.totalRequests - previousTotal) / elapsed).toFixed(1);
|
||||
document.getElementById('requestRate').textContent = `${rate} req/min`;
|
||||
}
|
||||
previousTotal = data.totalRequests;
|
||||
previousTime = now;
|
||||
|
||||
const successRate = data.totalRequests > 0
|
||||
? ((data.successRequests / data.totalRequests) * 100).toFixed(1)
|
||||
: 0;
|
||||
document.getElementById('successRate').textContent = `${successRate}% success rate`;
|
||||
|
||||
const errorRate = data.totalRequests > 0
|
||||
? ((data.errorRequests / data.totalRequests) * 100).toFixed(1)
|
||||
: 0;
|
||||
document.getElementById('errorRate').textContent = `${errorRate}% error rate`;
|
||||
|
||||
// Response time chart
|
||||
renderChart(data.lastResponseTimes || []);
|
||||
|
||||
// Endpoints
|
||||
renderEndpoints(data.requestsPerEndpoint || {});
|
||||
|
||||
// Uptime
|
||||
renderUptime(data.uptime || 0);
|
||||
|
||||
// Status codes
|
||||
renderStatusCodes(data.statusCodes || {});
|
||||
}
|
||||
|
||||
function setOffline(error) {
|
||||
const badge = document.getElementById('statusBadge');
|
||||
const statusText = document.getElementById('statusText');
|
||||
badge.className = 'status-badge offline';
|
||||
statusText.textContent = 'OFFLINE';
|
||||
document.getElementById('lastUpdated').textContent =
|
||||
new Date().toLocaleTimeString() + ` (${error})`;
|
||||
}
|
||||
|
||||
function animateValue(id, target) {
|
||||
const el = document.getElementById(id);
|
||||
const current = parseInt(el.textContent) || 0;
|
||||
if (current === target) return;
|
||||
const diff = target - current;
|
||||
const steps = 20;
|
||||
const increment = diff / steps;
|
||||
let step = 0;
|
||||
const timer = setInterval(() => {
|
||||
step++;
|
||||
el.textContent = Math.round(current + increment * step);
|
||||
if (step >= steps) {
|
||||
el.textContent = target;
|
||||
clearInterval(timer);
|
||||
}
|
||||
}, 20);
|
||||
}
|
||||
|
||||
function renderChart(times) {
|
||||
const container = document.getElementById('responseChart');
|
||||
if (times.length === 0) {
|
||||
container.innerHTML = '<div class="chart-empty">Waiting for data...</div>';
|
||||
return;
|
||||
}
|
||||
|
||||
const maxTime = Math.max(...times, 1);
|
||||
container.innerHTML = times.map((t, i) => {
|
||||
const height = Math.max((t / maxTime) * 180, 4);
|
||||
return `
|
||||
<div class="chart-bar-wrapper" title="${t}ms">
|
||||
<div class="chart-bar" style="height: ${height}px;"></div>
|
||||
<span class="chart-bar-label">${t}</span>
|
||||
</div>
|
||||
`;
|
||||
}).join('');
|
||||
}
|
||||
|
||||
function renderEndpoints(endpoints) {
|
||||
const list = document.getElementById('endpointList');
|
||||
const entries = Object.entries(endpoints);
|
||||
|
||||
if (entries.length === 0) {
|
||||
list.innerHTML = '<li class="endpoint-empty">No requests yet</li>';
|
||||
return;
|
||||
}
|
||||
|
||||
entries.sort((a, b) => b[1] - a[1]);
|
||||
list.innerHTML = entries.map(([name, count]) => `
|
||||
<li class="endpoint-item">
|
||||
<span class="endpoint-name">${name}</span>
|
||||
<span class="endpoint-count">${count}</span>
|
||||
</li>
|
||||
`).join('');
|
||||
}
|
||||
|
||||
function renderUptime(seconds) {
|
||||
const days = Math.floor(seconds / 86400);
|
||||
const hours = Math.floor((seconds % 86400) / 3600);
|
||||
const mins = Math.floor((seconds % 3600) / 60);
|
||||
const secs = Math.floor(seconds % 60);
|
||||
|
||||
document.getElementById('uptDays').textContent = days;
|
||||
document.getElementById('uptHours').textContent = hours;
|
||||
document.getElementById('uptMins').textContent = mins;
|
||||
document.getElementById('uptSecs').textContent = secs;
|
||||
}
|
||||
|
||||
function renderStatusCodes(codes) {
|
||||
let c2 = 0, c3 = 0, c4 = 0, c5 = 0;
|
||||
for (const [code, count] of Object.entries(codes)) {
|
||||
const num = parseInt(code);
|
||||
if (num >= 200 && num < 300) c2 += count;
|
||||
else if (num >= 300 && num < 400) c3 += count;
|
||||
else if (num >= 400 && num < 500) c4 += count;
|
||||
else if (num >= 500) c5 += count;
|
||||
}
|
||||
document.getElementById('code2xx').textContent = c2;
|
||||
document.getElementById('code3xx').textContent = c3;
|
||||
document.getElementById('code4xx').textContent = c4;
|
||||
document.getElementById('code5xx').textContent = c5;
|
||||
}
|
||||
|
||||
// Start polling
|
||||
fetchMetrics();
|
||||
setInterval(fetchMetrics, POLL_INTERVAL);
|
||||
</script>
|
||||
</body>
|
||||
</html>
|
||||
@@ -0,0 +1,387 @@
|
||||
# 🔴 Incident Postmortem Report — OOMKilled Application Crash
|
||||
|
||||
**Incident ID:** INC-2026-0726
|
||||
**Severity:** P1 — Critical
|
||||
**Duration:** 45 minutes
|
||||
**Date:** July 26, 2026
|
||||
**Author:** DevOps Team
|
||||
**Status:** Resolved
|
||||
|
||||
---
|
||||
|
||||
## 1. Executive Summary
|
||||
|
||||
On July 26, 2026, the **Ghayma REST API** application hosted on a cloud container platform experienced a **45-minute outage** caused by repeated **OOMKilled** (Out of Memory Killed) events. The container runtime terminated the application process multiple times after it exceeded its allocated memory limit. Each restart triggered the same memory spike, creating a crash loop that prevented the service from recovering.
|
||||
|
||||
**Impact:**
|
||||
- 100% service unavailability for 45 minutes
|
||||
- All API consumers (frontend clients, monitoring, health probes) received connection errors
|
||||
- Approximately **2,700 failed requests** during the outage window (estimated ~60 req/min baseline)
|
||||
- Container orchestrator marked the pod/container as `CrashLoopBackOff` after repeated restart failures
|
||||
|
||||
---
|
||||
|
||||
## 2. Incident Timeline
|
||||
|
||||
| Time (UTC) | Event |
|
||||
| :--- | :--- |
|
||||
| **11:00** | 🟢 Application running normally. Memory usage stable at ~120MB (limit: 256MB) |
|
||||
| **11:12** | 📈 Traffic spike begins — external batch job sends large payload requests to `POST /api/v1/items` |
|
||||
| **11:15** | ⚠️ Memory usage crosses **200MB** (78% of limit). No alerts triggered |
|
||||
| **11:18** | 🔴 Memory hits **256MB** limit. Kernel OOM killer terminates the container process (`OOMKilled`) |
|
||||
| **11:18** | 🔄 Container runtime automatically restarts the container (Restart #1) |
|
||||
| **11:20** | 🔴 Application starts, loads cached data into memory, immediately OOMKilled again (Restart #2) |
|
||||
| **11:20–11:45** | 🔁 **Crash loop** — container restarts 12 times. Orchestrator applies exponential backoff (`CrashLoopBackOff`) |
|
||||
| **11:32** | 📟 On-call engineer alerted via PagerDuty after health check failures exceed 10 minutes |
|
||||
| **11:38** | 🔍 Engineer identifies OOMKilled events in container logs and platform event stream |
|
||||
| **11:45** | 🛠️ Engineer increases memory limit from **256MB → 512MB** and deploys hotfix |
|
||||
| **11:48** | 🔄 Application restarts successfully. Memory stabilizes at ~180MB |
|
||||
| **11:50** | 📉 Batch job completes. Traffic returns to normal levels |
|
||||
| **12:03** | 🟢 Service fully confirmed stable. Incident resolved |
|
||||
|
||||
**Total downtime:** 45 minutes (11:18 — 12:03 UTC)
|
||||
|
||||
---
|
||||
|
||||
## 3. Root Cause Analysis
|
||||
|
||||
### 3.1 Direct Cause
|
||||
|
||||
The container was configured with a **memory limit of 256MB**, which was insufficient to handle traffic spikes. When an external batch job sent a burst of large `POST` requests with sizable JSON payloads, the Node.js process accumulated in-memory data (parsed request bodies, in-memory item storage, response buffers) beyond the container's limit.
|
||||
|
||||
The Linux kernel's **OOM Killer** terminated the process when it attempted to allocate memory beyond the `256MB` cgroup limit.
|
||||
|
||||
### 3.2 Contributing Factors
|
||||
|
||||
```
|
||||
┌──────────────────────────────────────────────────────────────────┐
|
||||
│ ROOT CAUSE BREAKDOWN │
|
||||
├──────────────────────────────────────────────────────────────────┤
|
||||
│ │
|
||||
│ 1. INSUFFICIENT MEMORY LIMIT │
|
||||
│ └─ Container limited to 256MB (too tight for Node.js) │
|
||||
│ │
|
||||
│ 2. NO MEMORY-AWARE AUTO-SCALING │
|
||||
│ └─ Only 1 replica running, no HPA/scaling policy │
|
||||
│ │
|
||||
│ 3. UNBOUNDED IN-MEMORY DATA STORE │
|
||||
│ └─ Items array grows indefinitely with no cap │
|
||||
│ │
|
||||
│ 4. NO MEMORY ALERTS / EARLY WARNING │
|
||||
│ └─ Alert only triggered after 10 min of health failures │
|
||||
│ │
|
||||
│ 5. NO REQUEST PAYLOAD SIZE LIMIT │
|
||||
│ └─ Express accepted arbitrarily large JSON bodies │
|
||||
│ │
|
||||
└──────────────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
### 3.3 Why Did Restarts Fail to Recover?
|
||||
|
||||
Each restart reloaded the same conditions:
|
||||
1. Container starts → Node.js initializes (~80MB baseline)
|
||||
2. Pending queued requests immediately hit the API
|
||||
3. Large payloads + in-memory storage push memory past the limit
|
||||
4. OOMKilled again within seconds
|
||||
5. Orchestrator enters **CrashLoopBackOff** with increasing delays between restarts
|
||||
|
||||
---
|
||||
|
||||
## 4. Remediation Actions Taken
|
||||
|
||||
| Action | Status |
|
||||
| :--- | :--- |
|
||||
| Increased container memory limit to 512MB | ✅ Done (hotfix) |
|
||||
| Restarted container with new limits | ✅ Done |
|
||||
| Confirmed service recovery and stability | ✅ Done |
|
||||
|
||||
---
|
||||
|
||||
## 5. Recommendations & Prevention
|
||||
|
||||
### 5.1 Application-Level Fixes
|
||||
|
||||
#### A. Limit Request Payload Size
|
||||
```javascript
|
||||
// In src/index.js — restrict incoming JSON body size
|
||||
app.use(express.json({ limit: '1mb' }));
|
||||
```
|
||||
> Prevents a single large request from consuming excessive memory.
|
||||
|
||||
#### B. Cap In-Memory Data Store
|
||||
```javascript
|
||||
// Limit the items array to prevent unbounded growth
|
||||
const MAX_ITEMS = 1000;
|
||||
|
||||
app.post('/api/v1/items', (req, res) => {
|
||||
if (items.length >= MAX_ITEMS) {
|
||||
return res.status(429).json({
|
||||
success: false,
|
||||
error: 'Maximum item limit reached'
|
||||
});
|
||||
}
|
||||
// ... rest of handler
|
||||
});
|
||||
```
|
||||
|
||||
#### C. Set Node.js Memory Ceiling
|
||||
```dockerfile
|
||||
# In Dockerfile — set explicit V8 heap limit
|
||||
CMD ["node", "--max-old-space-size=384", "src/index.js"]
|
||||
```
|
||||
> Ensures Node.js garbage collector runs more aggressively before hitting the container limit.
|
||||
|
||||
#### D. Use External Storage for Production
|
||||
Replace the in-memory `items` array with a proper database (Redis, PostgreSQL, MongoDB) so application memory remains constant regardless of data volume.
|
||||
|
||||
---
|
||||
|
||||
### 5.2 Container & Infrastructure Fixes
|
||||
|
||||
#### A. Set Proper Resource Requests & Limits
|
||||
|
||||
```yaml
|
||||
# Kubernetes Deployment example
|
||||
resources:
|
||||
requests:
|
||||
memory: "256Mi" # Guaranteed minimum
|
||||
cpu: "100m"
|
||||
limits:
|
||||
memory: "512Mi" # Hard ceiling
|
||||
cpu: "500m"
|
||||
```
|
||||
|
||||
> [!IMPORTANT]
|
||||
> **Rule of thumb:** Set `limits.memory` to at least **2x** the average working set. Set `requests.memory` to the **steady-state average**.
|
||||
|
||||
#### B. Add Graceful Shutdown Handling
|
||||
```javascript
|
||||
process.on('SIGTERM', () => {
|
||||
console.log('SIGTERM received. Shutting down gracefully...');
|
||||
server.close(() => {
|
||||
process.exit(0);
|
||||
});
|
||||
});
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 6. Auto-Scaling Policy Design
|
||||
|
||||
### 6.1 Horizontal Pod Autoscaler (HPA) Policy
|
||||
|
||||
The following auto-scaling policy prevents a single container from being overwhelmed by distributing load across multiple replicas:
|
||||
|
||||
```yaml
|
||||
apiVersion: autoscaling/v2
|
||||
kind: HorizontalPodAutoscaler
|
||||
metadata:
|
||||
name: ghayma-api-hpa
|
||||
namespace: production
|
||||
spec:
|
||||
scaleTargetRef:
|
||||
apiVersion: apps/v1
|
||||
kind: Deployment
|
||||
name: ghayma-api
|
||||
|
||||
# Replica bounds
|
||||
minReplicas: 2 # Always run at least 2 for high availability
|
||||
maxReplicas: 10 # Cap to control costs
|
||||
|
||||
# Scaling metrics
|
||||
metrics:
|
||||
# Scale based on MEMORY usage (primary - prevents OOMKill)
|
||||
- type: Resource
|
||||
resource:
|
||||
name: memory
|
||||
target:
|
||||
type: Utilization
|
||||
averageUtilization: 70 # Scale up when memory > 70% of limit
|
||||
|
||||
# Scale based on CPU usage (secondary)
|
||||
- type: Resource
|
||||
resource:
|
||||
name: cpu
|
||||
target:
|
||||
type: Utilization
|
||||
averageUtilization: 75 # Scale up when CPU > 75%
|
||||
|
||||
# Scaling behavior (prevents flapping)
|
||||
behavior:
|
||||
scaleUp:
|
||||
stabilizationWindowSeconds: 30 # React quickly to spikes
|
||||
policies:
|
||||
- type: Pods
|
||||
value: 2 # Add up to 2 pods at a time
|
||||
periodSeconds: 60
|
||||
scaleDown:
|
||||
stabilizationWindowSeconds: 300 # Wait 5 min before scaling down
|
||||
policies:
|
||||
- type: Pods
|
||||
value: 1 # Remove 1 pod at a time
|
||||
periodSeconds: 120
|
||||
```
|
||||
|
||||
### 6.2 How This Policy Prevents Duplication of the Incident
|
||||
|
||||
```
|
||||
Traffic Spike Detected
|
||||
│
|
||||
▼
|
||||
┌───────────────────────┐
|
||||
│ Memory usage > 70% │──── YES ──▶ HPA adds new replicas (up to 10)
|
||||
│ on existing pods? │ Load balancer distributes traffic
|
||||
└───────────────────────┘ Memory per-pod stays under limit
|
||||
│ NO │
|
||||
▼ ▼
|
||||
Normal operation Spike absorbed across multiple pods
|
||||
No single pod reaches OOMKill threshold
|
||||
```
|
||||
|
||||
### 6.3 Cloud Platform Auto-Scaling (Non-Kubernetes)
|
||||
|
||||
For managed container platforms (e.g., Ghayma Cloud, AWS ECS, Google Cloud Run):
|
||||
|
||||
| Setting | Value | Rationale |
|
||||
| :--- | :--- | :--- |
|
||||
| **Min instances** | 2 | Avoid cold starts & single point of failure |
|
||||
| **Max instances** | 10 | Cost ceiling |
|
||||
| **Scale-up trigger** | Memory > 70% OR CPU > 75% OR Concurrent requests > 50 | Multi-signal scaling |
|
||||
| **Scale-up cooldown** | 30 seconds | React quickly to spikes |
|
||||
| **Scale-down cooldown** | 5 minutes | Prevent flapping during variable traffic |
|
||||
| **Memory per instance** | 512MB | 2x steady-state working set |
|
||||
|
||||
---
|
||||
|
||||
## 7. Early Detection — Monitoring & Alerting Strategy
|
||||
|
||||
### 7.1 Key Metrics to Monitor
|
||||
|
||||
| Metric | Source | Warning Threshold | Critical Threshold |
|
||||
| :--- | :--- | :--- | :--- |
|
||||
| **Container Memory Usage** | cAdvisor / Platform metrics | > 70% of limit | > 85% of limit |
|
||||
| **Container Restart Count** | Kubelet / Platform events | ≥ 1 restart in 5 min | ≥ 3 restarts in 10 min |
|
||||
| **OOMKilled Events** | Kernel / Container runtime | Any occurrence | N/A (always critical) |
|
||||
| **Response Time (p95)** | Application `/metrics` endpoint | > 500ms | > 2000ms |
|
||||
| **Error Rate (5xx)** | Load balancer / Application | > 1% | > 5% |
|
||||
| **Health Check Failures** | Platform health probe | 1 consecutive failure | 3 consecutive failures |
|
||||
| **Request Queue Depth** | Load balancer | > 50 pending | > 200 pending |
|
||||
|
||||
### 7.2 Alerting Rules (Prometheus / Grafana Example)
|
||||
|
||||
```yaml
|
||||
# Alert: Memory approaching container limit
|
||||
- alert: HighMemoryUsage
|
||||
expr: |
|
||||
(container_memory_usage_bytes{container="ghayma-api"}
|
||||
/ container_spec_memory_limit_bytes{container="ghayma-api"}) > 0.70
|
||||
for: 2m
|
||||
labels:
|
||||
severity: warning
|
||||
annotations:
|
||||
summary: "Ghayma API memory usage above 70%"
|
||||
description: "Container {{ $labels.pod }} memory at {{ $value | humanizePercentage }}"
|
||||
|
||||
# Alert: OOMKill detected (CRITICAL — immediate page)
|
||||
- alert: OOMKillDetected
|
||||
expr: |
|
||||
increase(kube_pod_container_status_restarts_total{container="ghayma-api"}[5m]) > 0
|
||||
and kube_pod_container_status_last_terminated_reason{container="ghayma-api"} == "OOMKilled"
|
||||
for: 0m
|
||||
labels:
|
||||
severity: critical
|
||||
annotations:
|
||||
summary: "OOMKill detected on Ghayma API"
|
||||
description: "Pod {{ $labels.pod }} was OOMKilled. Immediate investigation required."
|
||||
|
||||
# Alert: High error rate
|
||||
- alert: HighErrorRate
|
||||
expr: |
|
||||
rate(http_requests_total{status=~"5.."}[5m])
|
||||
/ rate(http_requests_total[5m]) > 0.05
|
||||
for: 3m
|
||||
labels:
|
||||
severity: critical
|
||||
annotations:
|
||||
summary: "Ghayma API error rate exceeds 5%"
|
||||
```
|
||||
|
||||
### 7.3 Monitoring Architecture
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────────────────────────┐
|
||||
│ MONITORING STACK │
|
||||
│ │
|
||||
│ ┌──────────┐ ┌─────────────┐ ┌──────────────────────┐ │
|
||||
│ │ Ghayma │───▶│ Prometheus │───▶│ Grafana Dashboard │ │
|
||||
│ │ API │ │ (scrapes │ │ (visualization + │ │
|
||||
│ │ /metrics │ │ /metrics) │ │ alerting rules) │ │
|
||||
│ └──────────┘ └──────┬──────┘ └──────────────────────┘ │
|
||||
│ │ │
|
||||
│ ▼ │
|
||||
│ ┌─────────────────┐ │
|
||||
│ │ Alert Manager │ │
|
||||
│ │ (routing & │ │
|
||||
│ │ deduplication)│ │
|
||||
│ └────────┬────────┘ │
|
||||
│ │ │
|
||||
│ ┌────────────┼─────────────┐ │
|
||||
│ ▼ ▼ ▼ │
|
||||
│ ┌──────────┐ ┌──────────┐ ┌───────────┐ │
|
||||
│ │ Slack │ │ PagerDuty│ │ Email │ │
|
||||
│ │ #alerts │ │ on-call │ │ team │ │
|
||||
│ └──────────┘ └──────────┘ └───────────┘ │
|
||||
└─────────────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
### 7.4 Cloud Platform Native Tools
|
||||
|
||||
| Cloud Platform | Monitoring Service | Key Feature for OOM Detection |
|
||||
| :--- | :--- | :--- |
|
||||
| **AWS ECS / Fargate** | CloudWatch Container Insights | `MemoryUtilization` metric + CloudWatch Alarms |
|
||||
| **Google Cloud Run** | Cloud Monitoring | `container/memory/utilization` + Alert Policies |
|
||||
| **Azure Container Apps** | Azure Monitor | `MemoryWorkingSet` + Metric Alerts |
|
||||
| **Kubernetes (any)** | Prometheus + Grafana | `container_memory_usage_bytes` + `kube_pod_container_status_last_terminated_reason` |
|
||||
|
||||
### 7.5 Recommended Grafana Dashboard Panels
|
||||
|
||||
1. **Memory Usage vs Limit** — Time series showing memory consumption relative to the container limit (with 70% and 85% threshold lines)
|
||||
2. **Container Restarts** — Counter panel showing restart events over time
|
||||
3. **Response Time Heatmap** — p50 / p95 / p99 latency distribution
|
||||
4. **Request Rate & Error Rate** — Dual-axis graph (total requests + 5xx rate)
|
||||
5. **Pod/Replica Count** — Shows HPA scaling activity over time
|
||||
|
||||
---
|
||||
|
||||
## 8. Lessons Learned
|
||||
|
||||
| # | Lesson | Action Item |
|
||||
| :--- | :--- | :--- |
|
||||
| 1 | Default memory limits were set too low without load testing | Conduct load tests before production deployment |
|
||||
| 2 | Single-replica deployment has no resilience | Always run **≥ 2 replicas** for production services |
|
||||
| 3 | Alert threshold (10 min) was too slow for a P1 outage | Reduce critical alert threshold to **2 minutes** |
|
||||
| 4 | In-memory data stores are dangerous without bounds | Use external databases or enforce collection size limits |
|
||||
| 5 | No runbook existed for OOMKilled incidents | Create and publish an OOMKilled response runbook |
|
||||
|
||||
---
|
||||
|
||||
## 9. Action Items Tracker
|
||||
|
||||
| # | Action Item | Owner | Priority | Deadline | Status |
|
||||
| :--- | :--- | :--- | :--- | :--- | :--- |
|
||||
| 1 | Increase container memory limit to 512MB | DevOps | P0 | Done | ✅ |
|
||||
| 2 | Add `express.json({ limit: '1mb' })` payload limit | Backend | P1 | +1 day | ⬜ |
|
||||
| 3 | Implement HPA with memory-based scaling | DevOps | P1 | +3 days | ⬜ |
|
||||
| 4 | Set up Prometheus memory alerts (70% / 85%) | DevOps | P1 | +3 days | ⬜ |
|
||||
| 5 | Add OOMKilled alert rule (immediate paging) | DevOps | P0 | +1 day | ⬜ |
|
||||
| 6 | Run minimum 2 replicas in production | DevOps | P1 | +1 day | ⬜ |
|
||||
| 7 | Replace in-memory store with Redis/DB | Backend | P2 | +1 week | ⬜ |
|
||||
| 8 | Conduct load testing with realistic traffic | QA | P2 | +2 weeks | ⬜ |
|
||||
| 9 | Create OOMKilled incident runbook | DevOps | P2 | +1 week | ⬜ |
|
||||
| 10 | Build Grafana dashboard for container metrics | DevOps | P2 | +1 week | ⬜ |
|
||||
|
||||
---
|
||||
|
||||
> **Report prepared by:** DevOps Team
|
||||
> **Review date:** July 26, 2026
|
||||
> **Next review:** August 2, 2026 (verify all P0/P1 items completed)
|
||||
@@ -0,0 +1,218 @@
|
||||
name: CI/CD Pipeline — Build, Test & Deploy to Ghayma Cloud
|
||||
|
||||
on:
|
||||
push:
|
||||
branches: [main]
|
||||
pull_request:
|
||||
branches: [main]
|
||||
|
||||
env:
|
||||
REGISTRY: registry.ghaymah.systems
|
||||
IMAGE_NAME: ghayma-api
|
||||
STAGING_APP: ghayma-api-staging
|
||||
PRODUCTION_APP: ghayma-api-production
|
||||
|
||||
jobs:
|
||||
# ──────────────────────────────────────────────
|
||||
# Job 1: Build & Test
|
||||
# ──────────────────────────────────────────────
|
||||
build-and-test:
|
||||
name: 🔨 Build & Test
|
||||
runs-on: ubuntu-latest
|
||||
|
||||
steps:
|
||||
- name: Checkout code
|
||||
uses: actions/checkout@v4
|
||||
|
||||
- name: Setup Node.js
|
||||
uses: actions/setup-node@v4
|
||||
with:
|
||||
node-version: '20'
|
||||
cache: 'npm'
|
||||
|
||||
- name: Install dependencies
|
||||
run: npm ci
|
||||
|
||||
- name: Run syntax check
|
||||
run: node -c src/index.js
|
||||
|
||||
- name: Run health check test
|
||||
run: |
|
||||
# Start the server in the background
|
||||
node src/index.js &
|
||||
SERVER_PID=$!
|
||||
sleep 3
|
||||
|
||||
# Test health endpoint
|
||||
HTTP_STATUS=$(curl -s -o /dev/null -w "%{http_code}" http://localhost:3000/health)
|
||||
if [ "$HTTP_STATUS" -ne 200 ]; then
|
||||
echo "❌ Health check failed with status $HTTP_STATUS"
|
||||
kill $SERVER_PID
|
||||
exit 1
|
||||
fi
|
||||
|
||||
echo "✅ Health check passed (HTTP $HTTP_STATUS)"
|
||||
kill $SERVER_PID
|
||||
|
||||
# ──────────────────────────────────────────────
|
||||
# Job 2: Build Docker Image & Push to Registry
|
||||
# ──────────────────────────────────────────────
|
||||
docker-build-push:
|
||||
name: 🐳 Build & Push Docker Image
|
||||
runs-on: ubuntu-latest
|
||||
needs: build-and-test
|
||||
|
||||
outputs:
|
||||
image-tag: ${{ steps.meta.outputs.tags }}
|
||||
image-digest: ${{ steps.build-push.outputs.digest }}
|
||||
|
||||
steps:
|
||||
- name: Checkout code
|
||||
uses: actions/checkout@v4
|
||||
|
||||
- name: Log in to Ghayma Container Registry
|
||||
uses: docker/login-action@v3
|
||||
with:
|
||||
registry: ${{ env.REGISTRY }}
|
||||
username: ${{ secrets.GHAYMAH_REGISTRY_USERNAME }}
|
||||
password: ${{ secrets.GHAYMAH_REGISTRY_PASSWORD }}
|
||||
|
||||
- name: Extract Docker metadata (tags & labels)
|
||||
id: meta
|
||||
uses: docker/metadata-action@v5
|
||||
with:
|
||||
images: ${{ env.REGISTRY }}/${{ secrets.GHAYMAH_REGISTRY_USERNAME }}/${{ env.IMAGE_NAME }}
|
||||
tags: |
|
||||
type=sha,prefix=
|
||||
type=ref,event=branch
|
||||
type=raw,value=latest,enable={{is_default_branch}}
|
||||
|
||||
- name: Build and push Docker image
|
||||
id: build-push
|
||||
uses: docker/build-push-action@v5
|
||||
with:
|
||||
context: .
|
||||
push: true
|
||||
tags: ${{ steps.meta.outputs.tags }}
|
||||
labels: ${{ steps.meta.outputs.labels }}
|
||||
|
||||
- name: Print image details
|
||||
run: |
|
||||
echo "✅ Image pushed successfully"
|
||||
echo "📦 Tags: ${{ steps.meta.outputs.tags }}"
|
||||
echo "🔒 Digest: ${{ steps.build-push.outputs.digest }}"
|
||||
|
||||
# ──────────────────────────────────────────────
|
||||
# Job 3: Deploy to Staging (automatic)
|
||||
# ──────────────────────────────────────────────
|
||||
deploy-staging:
|
||||
name: 🟡 Deploy to Staging
|
||||
runs-on: ubuntu-latest
|
||||
needs: docker-build-push
|
||||
if: github.ref == 'refs/heads/main'
|
||||
environment:
|
||||
name: staging
|
||||
url: https://staging.ghaymah.systems
|
||||
|
||||
steps:
|
||||
- name: Install Ghayma CLI
|
||||
run: |
|
||||
curl -fsSL https://cli.ghaymah.systems/install.sh | bash
|
||||
echo "$HOME/.ghaymah/bin" >> $GITHUB_PATH
|
||||
|
||||
- name: Authenticate with Ghayma Cloud
|
||||
run: |
|
||||
ghaymah auth login \
|
||||
--token ${{ secrets.GHAYMAH_API_TOKEN }}
|
||||
|
||||
- name: Deploy to Staging environment
|
||||
run: |
|
||||
ghaymah app deploy ${{ env.STAGING_APP }} \
|
||||
--image ${{ env.REGISTRY }}/${{ secrets.GHAYMAH_REGISTRY_USERNAME }}/${{ env.IMAGE_NAME }}:latest \
|
||||
--env NODE_ENV=staging \
|
||||
--env PORT=3000 \
|
||||
--region me-central-1 \
|
||||
--wait
|
||||
|
||||
- name: Verify Staging deployment
|
||||
run: |
|
||||
sleep 10
|
||||
HTTP_STATUS=$(curl -s -o /dev/null -w "%{http_code}" https://staging.ghaymah.systems/health)
|
||||
if [ "$HTTP_STATUS" -ne 200 ]; then
|
||||
echo "❌ Staging health check failed (HTTP $HTTP_STATUS)"
|
||||
exit 1
|
||||
fi
|
||||
echo "✅ Staging deployment verified (HTTP $HTTP_STATUS)"
|
||||
|
||||
# ──────────────────────────────────────────────
|
||||
# Job 4: Manual Approval Gate
|
||||
# ──────────────────────────────────────────────
|
||||
# This job uses a GitHub Environment called
|
||||
# "production-approval" with "Required reviewers"
|
||||
# protection rule. The workflow PAUSES here and
|
||||
# waits for a designated team member to approve.
|
||||
# ──────────────────────────────────────────────
|
||||
manual-approval:
|
||||
name: ⏸️ Manual Approval Gate
|
||||
runs-on: ubuntu-latest
|
||||
needs: deploy-staging
|
||||
if: github.ref == 'refs/heads/main'
|
||||
environment:
|
||||
name: production-approval # Configure "Required reviewers" on this environment
|
||||
|
||||
steps:
|
||||
- name: Approval checkpoint
|
||||
run: |
|
||||
echo "✅ Production deployment has been approved!"
|
||||
echo "👤 Approved by: ${{ github.actor }}"
|
||||
echo "📦 Image: ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}:latest"
|
||||
echo "⏰ Approved at: $(date -u '+%Y-%m-%d %H:%M:%S UTC')"
|
||||
|
||||
# ──────────────────────────────────────────────
|
||||
# Job 5: Deploy to Production (after approval)
|
||||
# ──────────────────────────────────────────────
|
||||
deploy-production:
|
||||
name: 🟢 Deploy to Production
|
||||
runs-on: ubuntu-latest
|
||||
needs: manual-approval
|
||||
if: github.ref == 'refs/heads/main'
|
||||
environment:
|
||||
name: production
|
||||
url: https://api.ghaymah.systems
|
||||
|
||||
steps:
|
||||
- name: Install Ghayma CLI
|
||||
run: |
|
||||
curl -fsSL https://cli.ghaymah.systems/install.sh | bash
|
||||
echo "$HOME/.ghaymah/bin" >> $GITHUB_PATH
|
||||
|
||||
- name: Authenticate with Ghayma Cloud
|
||||
run: |
|
||||
ghaymah auth login \
|
||||
--token ${{ secrets.GHAYMAH_API_TOKEN }}
|
||||
|
||||
- name: Deploy to Production environment
|
||||
run: |
|
||||
ghaymah app deploy ${{ env.PRODUCTION_APP }} \
|
||||
--image ${{ env.REGISTRY }}/${{ secrets.GHAYMAH_REGISTRY_USERNAME }}/${{ env.IMAGE_NAME }}:latest \
|
||||
--env NODE_ENV=production \
|
||||
--env PORT=3000 \
|
||||
--region me-central-1 \
|
||||
--replicas 2 \
|
||||
--wait
|
||||
|
||||
- name: Verify Production deployment
|
||||
run: |
|
||||
sleep 10
|
||||
HTTP_STATUS=$(curl -s -o /dev/null -w "%{http_code}" https://api.ghaymah.systems/health)
|
||||
if [ "$HTTP_STATUS" -ne 200 ]; then
|
||||
echo "❌ Production health check failed (HTTP $HTTP_STATUS)"
|
||||
exit 1
|
||||
fi
|
||||
echo "✅ Production deployment verified (HTTP $HTTP_STATUS)"
|
||||
|
||||
- name: Notify success
|
||||
run: |
|
||||
echo "🚀 Production deployment complete!"
|
||||
echo "📦 Image: ${{ env.REGISTRY }}/${{ secrets.GHAYMAH_REGISTRY_USERNAME }}/${{ env.IMAGE_NAME }}:latest"
|
||||
echo "🌐 URL: https://api.ghaymah.systems"
|
||||
Binary file not shown.
|
After Width: | Height: | Size: 5.8 MiB |
@@ -0,0 +1,648 @@
|
||||
# Scalability & Load Distribution — Architecture Report
|
||||
|
||||
**Project:** Ghayma REST API
|
||||
**Platform:** ghaymah.systems (Ghayma Cloud)
|
||||
**Target Load:** 15,000 requests/second
|
||||
**Date:** July 26, 2026
|
||||
|
||||
---
|
||||
|
||||
## Table of Contents
|
||||
|
||||
1. [Architecture Diagram — 15,000 req/s](#1-architecture-diagram--15000-reqs)
|
||||
2. [Container Capacity Calculation](#2-container-capacity-calculation)
|
||||
3. [Cold Start Strategy for New Containers](#3-cold-start-strategy-for-new-containers)
|
||||
4. [Ghayma Block Storage for Stateful Workloads](#4-ghayma-block-storage-for-stateful-workloads)
|
||||
|
||||
---
|
||||
|
||||
## 1. Architecture Diagram — 15,000 req/s
|
||||
|
||||
### High-Level System Architecture
|
||||
|
||||
```
|
||||
15,000 req/s
|
||||
│
|
||||
▼
|
||||
┌───────────────────────┐
|
||||
│ DNS / CDN │
|
||||
│ (dns.ghaymah.systems)│
|
||||
└───────────┬───────────┘
|
||||
│
|
||||
▼
|
||||
┌───────────────────────┐
|
||||
│ LOAD BALANCER │
|
||||
│ (Layer 7 - HTTPS) │
|
||||
│ │
|
||||
│ • SSL Termination │
|
||||
│ • Health Checks │
|
||||
│ • Round Robin / │
|
||||
│ Least Connections │
|
||||
│ • Rate Limiting │
|
||||
└───────────┬───────────┘
|
||||
│
|
||||
┌────────────────────┼────────────────────┐
|
||||
│ │ │
|
||||
▼ ▼ ▼
|
||||
┌─────────────┐ ┌─────────────┐ ┌─────────────┐
|
||||
│ Zone A │ │ Zone B │ │ Zone C │
|
||||
│ (Rack/AZ 1) │ │ (Rack/AZ 2) │ │ (Rack/AZ 3) │
|
||||
└──────┬──────┘ └──────┬──────┘ └──────┬──────┘
|
||||
│ │ │
|
||||
┌──────┴──────┐ ┌──────┴──────┐ ┌──────┴──────┐
|
||||
│ ┌─────────┐ │ │ ┌─────────┐ │ │ ┌─────────┐ │
|
||||
│ │Container│ │ │ │Container│ │ │ │Container│ │
|
||||
│ │ 1-15 │ │ │ │ 16-30 │ │ │ │ 31-44 │ │
|
||||
│ │ (15x) │ │ │ │ (15x) │ │ │ │ (14x) │ │
|
||||
│ └─────────┘ │ │ └─────────┘ │ │ └─────────┘ │
|
||||
└──────┬──────┘ └──────┬──────┘ └──────┬──────┘
|
||||
│ │ │
|
||||
└───────────────────┼───────────────────┘
|
||||
│
|
||||
┌─────────────────┼─────────────────┐
|
||||
│ │ │
|
||||
▼ ▼ ▼
|
||||
┌──────────────┐ ┌──────────────┐ ┌──────────────┐
|
||||
│ Database │ │ Redis │ │ Block Storage│
|
||||
│ (Primary + │ │ (Session & │ │ (Persistent │
|
||||
│ Replicas) │ │ Cache) │ │ Volumes) │
|
||||
└──────────────┘ └──────────────┘ └──────────────┘
|
||||
```
|
||||
|
||||
### Detailed Component Architecture
|
||||
|
||||
```
|
||||
┌──────────────────────────────────────────────────────────────────────────────────────┐
|
||||
│ │
|
||||
│ GHAYMA CLOUD PLATFORM │
|
||||
│ │
|
||||
│ ┌────────────────────────────────────────────────────────────────────────────────┐ │
|
||||
│ │ NETWORKING LAYER │ │
|
||||
│ │ │ │
|
||||
│ │ Internet ──▶ [DNS] ──▶ [CDN/WAF] ──▶ [Load Balancer (L7)] │ │
|
||||
│ │ │ │ │
|
||||
│ │ ┌────────────────────┤ │ │
|
||||
│ │ │ Health Checks │ │ │
|
||||
│ │ │ GET /health (10s) │ │ │
|
||||
│ │ │ Threshold: 3 fails │ │ │
|
||||
│ │ └────────────────────┘ │ │
|
||||
│ └────────────────────────────────┬───────────────────────────────────────────────┘ │
|
||||
│ │ │
|
||||
│ ┌────────────────────────────────┼───────────────────────────────────────────────┐ │
|
||||
│ │ COMPUTE LAYER (44 Containers) │ │
|
||||
│ │ │ │
|
||||
│ │ ┌─────────┐ ┌─────────┐ ┌─────────┐ ┌─────────┐ ┌─────────┐ │ │
|
||||
│ │ │ C-01 │ │ C-02 │ │ C-03 │ │ C-04 │ ... │ C-44 │ │ │
|
||||
│ │ │ 500r/s │ │ 500r/s │ │ 500r/s │ │ 500r/s │ │ 500r/s │ │ │
|
||||
│ │ │ 512MB │ │ 512MB │ │ 512MB │ │ 512MB │ │ 512MB │ │ │
|
||||
│ │ │ 0.5 CPU │ │ 0.5 CPU │ │ 0.5 CPU │ │ 0.5 CPU │ │ 0.5 CPU │ │ │
|
||||
│ │ └────┬────┘ └────┬────┘ └────┬────┘ └────┬────┘ └────┬────┘ │ │
|
||||
│ │ └────────────┴────────────┴────────────┴──── ... ───────┘ │ │
|
||||
│ │ │ │ │
|
||||
│ │ ┌──────────────────────────────────────────────────────────────────┐ │ │
|
||||
│ │ │ AUTO-SCALER (HPA) │ │ │
|
||||
│ │ │ Min: 44 | Max: 66 | Scale at: CPU>75% or Mem>70% │ │ │
|
||||
│ │ └──────────────────────────────────────────────────────────────────┘ │ │
|
||||
│ └────────────────────────────────┬───────────────────────────────────────────────┘ │
|
||||
│ │ │
|
||||
│ ┌────────────────────────────────┼───────────────────────────────────────────────┐ │
|
||||
│ │ DATA LAYER │ │
|
||||
│ │ │ │
|
||||
│ │ ┌──────────────────┐ ┌─────────────┐ ┌──────────────────────────────┐ │ │
|
||||
│ │ │ PostgreSQL DB │ │ Redis │ │ Ghayma Block Storage │ │ │
|
||||
│ │ │ Primary + 2 │ │ Cluster │ │ (Persistent Volumes) │ │ │
|
||||
│ │ │ Read Replicas │ │ (Cache + │ │ • Database data (100GB) │ │ │
|
||||
│ │ │ │ │ Sessions) │ │ • Logs & uploads (50GB) │ │ │
|
||||
│ │ └──────────────────┘ └─────────────┘ └──────────────────────────────┘ │ │
|
||||
│ └────────────────────────────────────────────────────────────────────────────────┘ │
|
||||
│ │
|
||||
│ ┌────────────────────────────────────────────────────────────────────────────────┐ │
|
||||
│ │ OBSERVABILITY LAYER │ │
|
||||
│ │ │ │
|
||||
│ │ [Prometheus] ──▶ [Grafana Dashboards] ──▶ [AlertManager] ──▶ [PagerDuty] │ │
|
||||
│ │ [Container Logs] ──▶ [Centralized Logging] ──▶ [Log Analytics] │ │
|
||||
│ └────────────────────────────────────────────────────────────────────────────────┘ │
|
||||
│ │
|
||||
└──────────────────────────────────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
### Load Balancer Distribution Strategy
|
||||
|
||||
```
|
||||
15,000 req/s incoming
|
||||
│
|
||||
┌────────┴────────┐
|
||||
│ LOAD BALANCER │
|
||||
│ │
|
||||
│ Algorithm: │
|
||||
│ Least │
|
||||
│ Connections │
|
||||
└────────┬────────┘
|
||||
│
|
||||
┌────────────────┼────────────────┐
|
||||
│ │ │
|
||||
Zone A (33%) Zone B (33%) Zone C (34%)
|
||||
~5,000 r/s ~5,000 r/s ~5,000 r/s
|
||||
15 containers 15 containers 14 containers
|
||||
~333 r/s each ~333 r/s each ~357 r/s each
|
||||
```
|
||||
|
||||
> Each container handles **~341 req/s on average** — well within the 500 req/s capacity, leaving the 30% safety margin intact.
|
||||
|
||||
---
|
||||
|
||||
## 2. Container Capacity Calculation
|
||||
|
||||
### Given Parameters
|
||||
|
||||
| Parameter | Value |
|
||||
| :--- | :--- |
|
||||
| **Total traffic** | 15,000 req/s |
|
||||
| **Capacity per container** | 500 req/s |
|
||||
| **Safety margin** | 30% |
|
||||
|
||||
### Step-by-Step Calculation
|
||||
|
||||
#### Step 1: Calculate Effective Capacity per Container
|
||||
|
||||
The 30% margin means each container should only be utilized at 70% of its max capacity:
|
||||
|
||||
```
|
||||
Effective capacity = Max capacity × (1 - Safety margin)
|
||||
Effective capacity = 500 req/s × (1 - 0.30)
|
||||
Effective capacity = 500 × 0.70
|
||||
Effective capacity = 350 req/s per container
|
||||
```
|
||||
|
||||
#### Step 2: Calculate Minimum Containers Required
|
||||
|
||||
```
|
||||
Minimum containers = Total traffic ÷ Effective capacity
|
||||
Minimum containers = 15,000 ÷ 350
|
||||
Minimum containers = 42.86
|
||||
Minimum containers = 43 containers (rounded up)
|
||||
```
|
||||
|
||||
#### Step 3: Add Availability Buffer (+1)
|
||||
|
||||
To handle single-container failures without breaching capacity:
|
||||
|
||||
```
|
||||
Recommended containers = 43 + 1 (failure tolerance)
|
||||
Recommended containers = 44 containers
|
||||
```
|
||||
|
||||
### Summary Table
|
||||
|
||||
| Metric | Value | Formula |
|
||||
| :--- | :--- | :--- |
|
||||
| **Total load** | 15,000 req/s | Given |
|
||||
| **Max capacity/container** | 500 req/s | Given |
|
||||
| **Safety margin** | 30% | Given |
|
||||
| **Effective capacity/container** | 350 req/s | 500 × 0.70 |
|
||||
| **Minimum containers** | 43 | ⌈15,000 ÷ 350⌉ |
|
||||
| **Recommended (with buffer)** | **44** | 43 + 1 |
|
||||
| **Actual utilization per container** | ~341 req/s | 15,000 ÷ 44 |
|
||||
| **Utilization percentage** | 68.2% | 341 ÷ 500 |
|
||||
| **Remaining headroom** | 31.8% | ✅ Exceeds 30% margin |
|
||||
|
||||
### Capacity Planning Visualization
|
||||
|
||||
```
|
||||
Container Capacity: 500 req/s
|
||||
├─────────────────────────────────────────────────┤
|
||||
0 500
|
||||
|
||||
Actual load per container: ~341 req/s (68.2%)
|
||||
├──────────────────────────────────┤
|
||||
0 341
|
||||
|
||||
Safety margin zone (30%):
|
||||
├──────────────────────────────────┤░░░░░░░░░░░░░░┤
|
||||
0 350 500
|
||||
▲
|
||||
Threshold
|
||||
(don't exceed)
|
||||
```
|
||||
|
||||
### Auto-Scaling Tiers
|
||||
|
||||
| Traffic Level | Containers Needed | Status |
|
||||
| :--- | :--- | :--- |
|
||||
| 5,000 req/s (low) | 15 | Minimum viable |
|
||||
| 10,000 req/s (moderate) | 30 | Normal scaling |
|
||||
| **15,000 req/s (target)** | **44** | **Design target** |
|
||||
| 20,000 req/s (spike) | 58 | Burst capacity |
|
||||
| 25,000 req/s (peak) | 72 | Maximum scaling |
|
||||
|
||||
### Resource Allocation per Container
|
||||
|
||||
| Resource | Value |
|
||||
| :--- | :--- |
|
||||
| Memory Request | 256 Mi |
|
||||
| Memory Limit | 512 Mi |
|
||||
| CPU Request | 250m (0.25 vCPU) |
|
||||
| CPU Limit | 500m (0.5 vCPU) |
|
||||
|
||||
### Total Cluster Resources (44 containers)
|
||||
|
||||
| Resource | Per Container | × 44 Containers | Total |
|
||||
| :--- | :--- | :--- | :--- |
|
||||
| **Memory** | 512 Mi | × 44 | **22 GB** |
|
||||
| **CPU** | 0.5 vCPU | × 44 | **22 vCPUs** |
|
||||
| **Throughput** | 350 req/s (effective) | × 44 | **15,400 req/s** |
|
||||
|
||||
---
|
||||
|
||||
## 3. Cold Start Strategy for New Containers
|
||||
|
||||
### What is a Cold Start?
|
||||
|
||||
A **cold start** occurs when a new container instance is created and must initialize before serving traffic. During this period:
|
||||
|
||||
- The container image is pulled from the registry
|
||||
- The Node.js runtime starts
|
||||
- Dependencies are loaded
|
||||
- The application connects to databases/caches
|
||||
- The first requests experience **higher latency**
|
||||
|
||||
```
|
||||
Cold Start Timeline (without optimization)
|
||||
──────────────────────────────────────────────────────
|
||||
│ Image Pull │ Runtime Init │ App Boot │ DB Connect │ Ready │
|
||||
0s 3s 4s 5s 7s 8s
|
||||
▲
|
||||
First request served
|
||||
```
|
||||
|
||||
### Cold Start Strategy — 5 Layers
|
||||
|
||||
#### Layer 1: Pre-Warmed Container Pool
|
||||
|
||||
Maintain a pool of **idle but ready** containers that can immediately accept traffic:
|
||||
|
||||
```
|
||||
Active Containers (44) Pre-Warmed Pool (4-6)
|
||||
┌────┬────┬────┬─ ... ─┬────┐ ┌────┬────┬────┬────┐
|
||||
│ C1 │ C2 │ C3 │ │C44 │ │ W1 │ W2 │ W3 │ W4 │
|
||||
│BUSY│BUSY│BUSY│ │BUSY│ │IDLE│IDLE│IDLE│IDLE│
|
||||
└────┴────┴────┴─ ... ─┴────┘ └────┴────┴────┴────┘
|
||||
│
|
||||
Traffic spike detected
|
||||
│
|
||||
▼
|
||||
W1, W2 immediately start
|
||||
serving (0s cold start)
|
||||
```
|
||||
|
||||
**Implementation:**
|
||||
```yaml
|
||||
# HPA with pre-warmed minimum
|
||||
apiVersion: autoscaling/v2
|
||||
kind: HorizontalPodAutoscaler
|
||||
spec:
|
||||
minReplicas: 48 # 44 active + 4 pre-warmed
|
||||
maxReplicas: 72
|
||||
```
|
||||
|
||||
#### Layer 2: Optimized Docker Image (Fast Pulls)
|
||||
|
||||
Reduce image pull time by using lightweight images:
|
||||
|
||||
```dockerfile
|
||||
# ✅ GOOD: Alpine-based image (~150MB, pulls in ~2s)
|
||||
FROM node:20-alpine
|
||||
|
||||
# ❌ BAD: Full Debian image (~1GB, pulls in ~12s)
|
||||
FROM node:20
|
||||
```
|
||||
|
||||
| Image Base | Size | Pull Time | Cold Start Impact |
|
||||
| :--- | :--- | :--- | :--- |
|
||||
| `node:20` (Debian) | ~1 GB | ~12s | Very slow |
|
||||
| `node:20-slim` | ~250 MB | ~5s | Moderate |
|
||||
| `node:20-alpine` | ~150 MB | ~2s | ✅ **Fast** |
|
||||
|
||||
#### Layer 3: Application-Level Readiness
|
||||
|
||||
Implement a **readiness probe** that only marks the container as "ready" after it has fully initialized:
|
||||
|
||||
```javascript
|
||||
// In src/index.js
|
||||
let isReady = false;
|
||||
|
||||
// Readiness check - only returns 200 when app is fully initialized
|
||||
app.get('/ready', (req, res) => {
|
||||
if (isReady) {
|
||||
res.status(200).json({ ready: true });
|
||||
} else {
|
||||
res.status(503).json({ ready: false, message: 'Warming up...' });
|
||||
}
|
||||
});
|
||||
|
||||
// App startup sequence
|
||||
async function initialize() {
|
||||
// 1. Connect to database
|
||||
await connectDatabase();
|
||||
|
||||
// 2. Connect to Redis cache
|
||||
await connectRedis();
|
||||
|
||||
// 3. Pre-load frequently accessed data into memory
|
||||
await preloadCache();
|
||||
|
||||
// 4. Mark as ready ONLY after everything is initialized
|
||||
isReady = true;
|
||||
console.log('✅ Application fully initialized and ready');
|
||||
}
|
||||
|
||||
app.listen(PORT, async () => {
|
||||
console.log(`Server starting on port ${PORT}...`);
|
||||
await initialize();
|
||||
});
|
||||
```
|
||||
|
||||
```yaml
|
||||
# Kubernetes readiness probe
|
||||
readinessProbe:
|
||||
httpGet:
|
||||
path: /ready # Not /health — separate from liveness
|
||||
port: 3000
|
||||
initialDelaySeconds: 3
|
||||
periodSeconds: 5
|
||||
failureThreshold: 3 # Must pass 3 checks before receiving traffic
|
||||
```
|
||||
|
||||
> The load balancer **will not route traffic** to the container until `/ready` returns HTTP 200.
|
||||
|
||||
#### Layer 4: Proactive Scaling (Anticipate Demand)
|
||||
|
||||
Scale **before** the spike hits, not after:
|
||||
|
||||
```yaml
|
||||
# CronJob-based pre-scaling for known traffic patterns
|
||||
apiVersion: batch/v1
|
||||
kind: CronJob
|
||||
metadata:
|
||||
name: pre-scale-morning
|
||||
spec:
|
||||
# Scale up at 8:00 AM before morning traffic rush
|
||||
schedule: "0 8 * * 1-5"
|
||||
jobTemplate:
|
||||
spec:
|
||||
template:
|
||||
spec:
|
||||
containers:
|
||||
- name: scaler
|
||||
image: bitnami/kubectl:latest
|
||||
command:
|
||||
- kubectl
|
||||
- scale
|
||||
- deployment/ghayma-api
|
||||
- --replicas=50
|
||||
```
|
||||
|
||||
#### Layer 5: Connection Pooling & Warm-Up Requests
|
||||
|
||||
After a container starts, send **internal warm-up requests** to pre-establish connections:
|
||||
|
||||
```javascript
|
||||
async function warmUp() {
|
||||
console.log('🔥 Running warm-up sequence...');
|
||||
|
||||
// Pre-establish database connection pool
|
||||
await db.query('SELECT 1');
|
||||
|
||||
// Prime V8 JIT compiler with typical request patterns
|
||||
const warmUpPaths = ['/health', '/api/v1/items'];
|
||||
for (const path of warmUpPaths) {
|
||||
await fetch(`http://localhost:${PORT}${path}`);
|
||||
}
|
||||
|
||||
console.log('✅ Warm-up complete — ready for production traffic');
|
||||
}
|
||||
```
|
||||
|
||||
### Cold Start Strategy Summary
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────────────────────────┐
|
||||
│ COLD START MITIGATION LAYERS │
|
||||
│ │
|
||||
│ Layer 1: Pre-Warmed Pool → 0s cold start (instant) │
|
||||
│ Layer 2: Alpine Docker Image → 2s image pull (vs 12s) │
|
||||
│ Layer 3: Readiness Probes → No traffic until fully ready │
|
||||
│ Layer 4: Proactive Scaling → Scale before the spike │
|
||||
│ Layer 5: Warm-Up Requests → Pre-establish connections │
|
||||
│ │
|
||||
│ Combined Effect: New containers ready in ~3-5s │
|
||||
│ Without strategy: New containers ready in ~15-20s │
|
||||
└─────────────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 4. Ghayma Block Storage for Stateful Workloads
|
||||
|
||||
### What is Ghayma Block Storage?
|
||||
|
||||
**Ghayma Block Storage** is a persistent, network-attached block storage service provided by the Ghayma Cloud platform. It provides raw storage volumes that can be attached to containers, similar to a virtual hard drive.
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────────────────────────┐
|
||||
│ BLOCK STORAGE vs CONTAINER STORAGE │
|
||||
│ │
|
||||
│ Container (Ephemeral) Block Storage (Persistent) │
|
||||
│ ┌──────────────────┐ ┌──────────────────────┐ │
|
||||
│ │ Container FS │ │ Ghayma Block Volume │ │
|
||||
│ │ │ │ │ │
|
||||
│ │ ❌ Data lost │ │ ✅ Data survives │ │
|
||||
│ │ when container│ │ container restarts│ │
|
||||
│ │ stops/crashes │ │ and redeployments │ │
|
||||
│ │ │ │ │ │
|
||||
│ │ ❌ Not shared │ │ ✅ Can be reattached │ │
|
||||
│ │ between │ │ to new containers │ │
|
||||
│ │ containers │ │ │ │
|
||||
│ └──────────────────┘ └──────────────────────┘ │
|
||||
│ │
|
||||
└─────────────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
### Why is Block Storage Needed?
|
||||
|
||||
Containers are **stateless and ephemeral** by design — when a container restarts, crashes, or is replaced during a deployment, all data written inside it is **lost**. This is a problem for:
|
||||
|
||||
| Workload Type | Data at Risk | Block Storage Solution |
|
||||
| :--- | :--- | :--- |
|
||||
| **Databases** (PostgreSQL, MySQL) | Table data, indexes, WAL logs | Persistent volume for `/var/lib/postgresql/data` |
|
||||
| **Cache** (Redis with persistence) | RDB/AOF snapshots | Volume for `/data` |
|
||||
| **File Uploads** | User-uploaded images, documents | Volume for `/app/uploads` |
|
||||
| **Application Logs** | Log files for audit/compliance | Volume for `/var/log/app` |
|
||||
| **Session Storage** | User session files | Volume for `/tmp/sessions` |
|
||||
|
||||
### Architecture: Block Storage with Containers
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────────────────────────────┐
|
||||
│ STATEFUL WORKLOAD ARCHITECTURE │
|
||||
│ │
|
||||
│ ┌───────────────────┐ ┌───────────────────────────────┐ │
|
||||
│ │ API Container │ │ Database Container │ │
|
||||
│ │ (Stateless) │ │ (Stateful — PostgreSQL) │ │
|
||||
│ │ │──────▶│ │ │
|
||||
│ │ No volume │ │ ┌─────────────────────┐ │ │
|
||||
│ │ needed │ │ │ /var/lib/postgresql │ │ │
|
||||
│ │ │ │ │ (mount point) │ │ │
|
||||
│ └───────────────────┘ │ └──────────┬──────────┘ │ │
|
||||
│ │ │ │ │
|
||||
│ └──────────────┼───────────────┘ │
|
||||
│ │ │
|
||||
│ ┌─────────▼──────────┐ │
|
||||
│ │ GHAYMA BLOCK │ │
|
||||
│ │ STORAGE VOLUME │ │
|
||||
│ │ │ │
|
||||
│ │ Name: db-data │ │
|
||||
│ │ Size: 100 GB │ │
|
||||
│ │ Type: SSD (gp3) │ │
|
||||
│ │ IOPS: 3,000 │ │
|
||||
│ │ Encrypted: Yes │ │
|
||||
│ │ │ │
|
||||
│ │ ✅ Persists across │ │
|
||||
│ │ restarts │ │
|
||||
│ │ ✅ Automatic │ │
|
||||
│ │ snapshots │ │
|
||||
│ │ ✅ Reattachable │ │
|
||||
│ └────────────────────┘ │
|
||||
│ │
|
||||
└────────────────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
### How to Use Ghayma Block Storage
|
||||
|
||||
#### Step 1: Create a Block Storage Volume
|
||||
|
||||
```bash
|
||||
# Using Ghayma CLI
|
||||
ghaymah storage create db-data \
|
||||
--size 100GB \
|
||||
--type ssd \
|
||||
--region me-central-1 \
|
||||
--encrypted
|
||||
```
|
||||
|
||||
#### Step 2: Attach to a Container / Deployment
|
||||
|
||||
```yaml
|
||||
# Kubernetes PersistentVolumeClaim using Ghayma storage class
|
||||
apiVersion: v1
|
||||
kind: PersistentVolumeClaim
|
||||
metadata:
|
||||
name: db-data-pvc
|
||||
spec:
|
||||
accessModes:
|
||||
- ReadWriteOnce # Single container read/write
|
||||
storageClassName: ghaymah-block-ssd
|
||||
resources:
|
||||
requests:
|
||||
storage: 100Gi
|
||||
---
|
||||
# Deployment using the volume
|
||||
apiVersion: apps/v1
|
||||
kind: Deployment
|
||||
metadata:
|
||||
name: postgres-db
|
||||
spec:
|
||||
replicas: 1 # Stateful — typically 1 primary
|
||||
template:
|
||||
spec:
|
||||
containers:
|
||||
- name: postgres
|
||||
image: postgres:16-alpine
|
||||
ports:
|
||||
- containerPort: 5432
|
||||
volumeMounts:
|
||||
- name: db-storage
|
||||
mountPath: /var/lib/postgresql/data
|
||||
env:
|
||||
- name: POSTGRES_PASSWORD
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
name: db-secret
|
||||
key: password
|
||||
volumes:
|
||||
- name: db-storage
|
||||
persistentVolumeClaim:
|
||||
claimName: db-data-pvc # Ghayma Block Storage volume
|
||||
```
|
||||
|
||||
#### Step 3: Verify the Volume
|
||||
|
||||
```bash
|
||||
# Check volume status
|
||||
ghaymah storage list
|
||||
|
||||
# Output:
|
||||
# NAME SIZE TYPE STATUS ATTACHED TO REGION
|
||||
# db-data 100GB SSD attached postgres-db-pod-0 me-central-1
|
||||
```
|
||||
|
||||
### Block Storage Features on Ghayma Cloud
|
||||
|
||||
| Feature | Description |
|
||||
| :--- | :--- |
|
||||
| **Persistence** | Data survives container restarts, crashes, and redeployments |
|
||||
| **SSD Performance** | Low-latency SSD storage with up to 16,000 IOPS |
|
||||
| **Encryption** | At-rest encryption (AES-256) for security compliance |
|
||||
| **Snapshots** | Point-in-time snapshots for backup and disaster recovery |
|
||||
| **Resizing** | Expand volume size without downtime |
|
||||
| **Reattachment** | Detach from one container and attach to another (for migration) |
|
||||
| **Availability Zones** | Volume and container must be in the same zone |
|
||||
|
||||
### When to Use Block Storage vs Other Options
|
||||
|
||||
| Storage Type | Use Case | Persistence | Shared Access | Speed |
|
||||
| :--- | :--- | :--- | :--- | :--- |
|
||||
| **Container FS** | Temp files, build artifacts | ❌ Ephemeral | ❌ No | ⚡ Fastest |
|
||||
| **Ghayma Block Storage** | Databases, stateful apps, logs | ✅ Persistent | ❌ Single container | ⚡ Fast (SSD) |
|
||||
| **Object Storage (S3-like)** | Images, videos, backups | ✅ Persistent | ✅ Multi-container | 🐌 Moderate |
|
||||
| **Shared File Storage (NFS)** | Shared config, media libraries | ✅ Persistent | ✅ Multi-container | 🔄 Moderate |
|
||||
|
||||
### Data Lifecycle with Block Storage
|
||||
|
||||
```
|
||||
Container crashes or redeploys
|
||||
│
|
||||
▼
|
||||
┌───────────────────────────────────────────┐
|
||||
│ Old container is terminated │
|
||||
│ ❌ Container filesystem is destroyed │
|
||||
│ ✅ Block Storage volume is PRESERVED │
|
||||
└────────────────────┬──────────────────────┘
|
||||
│
|
||||
▼
|
||||
┌───────────────────────────────────────────┐
|
||||
│ New container starts │
|
||||
│ Block Storage volume is reattached │
|
||||
│ Data is immediately available at the │
|
||||
│ same mount path (/var/lib/postgresql) │
|
||||
│ │
|
||||
│ ✅ Zero data loss │
|
||||
│ ✅ Database resumes from exact state │
|
||||
└───────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Summary
|
||||
|
||||
| Requirement | Solution |
|
||||
| :--- | :--- |
|
||||
| **Architecture for 15K req/s** | 3-zone architecture with L7 load balancer, 44 containers, auto-scaler, and data layer |
|
||||
| **Container count** | **44 containers** (15,000 ÷ 350 effective req/s + 1 buffer) at 68.2% utilization |
|
||||
| **Cold start strategy** | 5-layer approach: pre-warmed pool, Alpine images, readiness probes, proactive scaling, warm-up requests |
|
||||
| **Block Storage** | Persistent SSD volumes for databases and stateful workloads that survive container restarts |
|
||||
|
||||
---
|
||||
|
||||
> **Document prepared for:** Ghayma Assessment — Scalability & Load Distribution section
|
||||
> **Last updated:** July 26, 2026
|
||||
@@ -0,0 +1,678 @@
|
||||
<!DOCTYPE html>
|
||||
<html lang="en">
|
||||
<head>
|
||||
<meta charset="UTF-8">
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1.0">
|
||||
<title>mithal.space — Monitoring Dashboard</title>
|
||||
<meta name="description" content="Real-time monitoring dashboard for mithal.space — uptime, latency, SSL, DNS">
|
||||
<link href="https://fonts.googleapis.com/css2?family=Inter:wght@300;400;500;600;700;800;900&display=swap" rel="stylesheet">
|
||||
<style>
|
||||
*, *::before, *::after { margin: 0; padding: 0; box-sizing: border-box; }
|
||||
|
||||
:root {
|
||||
--bg-primary: #06080f;
|
||||
--bg-secondary: #0c1120;
|
||||
--bg-card: rgba(12, 17, 32, 0.85);
|
||||
--bg-card-hover: rgba(18, 25, 45, 0.9);
|
||||
--border: rgba(255, 255, 255, 0.05);
|
||||
--border-hover: rgba(255, 255, 255, 0.1);
|
||||
--text-primary: #e8edf5;
|
||||
--text-secondary: #8892a8;
|
||||
--text-muted: #525d75;
|
||||
--green: #10b981;
|
||||
--green-glow: rgba(16, 185, 129, 0.12);
|
||||
--red: #ef4444;
|
||||
--red-glow: rgba(239, 68, 68, 0.12);
|
||||
--blue: #6366f1;
|
||||
--blue-glow: rgba(99, 102, 241, 0.12);
|
||||
--amber: #f59e0b;
|
||||
--amber-glow: rgba(245, 158, 11, 0.12);
|
||||
--purple: #a855f7;
|
||||
--purple-glow: rgba(168, 85, 247, 0.12);
|
||||
--cyan: #06b6d4;
|
||||
--cyan-glow: rgba(6, 182, 212, 0.12);
|
||||
--radius: 16px;
|
||||
--radius-sm: 10px;
|
||||
--radius-xs: 6px;
|
||||
}
|
||||
|
||||
body {
|
||||
font-family: 'Inter', -apple-system, sans-serif;
|
||||
background: var(--bg-primary);
|
||||
color: var(--text-primary);
|
||||
min-height: 100vh;
|
||||
overflow-x: hidden;
|
||||
}
|
||||
|
||||
body::before {
|
||||
content: '';
|
||||
position: fixed;
|
||||
inset: 0;
|
||||
background:
|
||||
radial-gradient(ellipse at 15% 20%, rgba(99, 102, 241, 0.06) 0%, transparent 60%),
|
||||
radial-gradient(ellipse at 85% 80%, rgba(168, 85, 247, 0.04) 0%, transparent 50%),
|
||||
radial-gradient(ellipse at 50% 50%, rgba(6, 182, 212, 0.03) 0%, transparent 50%);
|
||||
pointer-events: none;
|
||||
z-index: 0;
|
||||
}
|
||||
|
||||
.app { position: relative; z-index: 1; }
|
||||
|
||||
/* ── Header ── */
|
||||
.header {
|
||||
padding: 20px 32px;
|
||||
display: flex;
|
||||
align-items: center;
|
||||
justify-content: space-between;
|
||||
border-bottom: 1px solid var(--border);
|
||||
backdrop-filter: blur(24px);
|
||||
background: rgba(6, 8, 15, 0.7);
|
||||
position: sticky;
|
||||
top: 0;
|
||||
z-index: 100;
|
||||
}
|
||||
|
||||
.header-left { display: flex; align-items: center; gap: 14px; }
|
||||
|
||||
.logo {
|
||||
width: 38px; height: 38px;
|
||||
border-radius: 10px;
|
||||
background: linear-gradient(135deg, var(--blue), var(--purple));
|
||||
display: flex; align-items: center; justify-content: center;
|
||||
font-size: 16px; font-weight: 900; color: #fff;
|
||||
box-shadow: 0 4px 24px rgba(99, 102, 241, 0.3);
|
||||
}
|
||||
|
||||
.header-title { font-size: 18px; font-weight: 700; letter-spacing: -0.4px; }
|
||||
.header-title span { color: var(--text-muted); font-weight: 400; font-size: 14px; margin-left: 6px; }
|
||||
|
||||
.header-right { display: flex; align-items: center; gap: 14px; }
|
||||
|
||||
.live-badge {
|
||||
display: flex; align-items: center; gap: 7px;
|
||||
padding: 6px 14px; border-radius: 100px;
|
||||
font-size: 12px; font-weight: 600; letter-spacing: 0.4px;
|
||||
transition: all 0.4s ease;
|
||||
}
|
||||
.live-badge.up { background: var(--green-glow); color: var(--green); border: 1px solid rgba(16, 185, 129, 0.2); }
|
||||
.live-badge.down { background: var(--red-glow); color: var(--red); border: 1px solid rgba(239, 68, 68, 0.2); }
|
||||
.live-badge.unknown { background: rgba(255,255,255,0.05); color: var(--text-muted); border: 1px solid var(--border); }
|
||||
|
||||
.live-dot {
|
||||
width: 7px; height: 7px; border-radius: 50%; position: relative;
|
||||
}
|
||||
.live-badge.up .live-dot { background: var(--green); }
|
||||
.live-badge.down .live-dot { background: var(--red); }
|
||||
.live-badge.unknown .live-dot { background: var(--text-muted); }
|
||||
.live-dot::after {
|
||||
content: '';
|
||||
position: absolute; top: -3px; left: -3px;
|
||||
width: 13px; height: 13px; border-radius: 50%;
|
||||
animation: pulse 2s ease-out infinite;
|
||||
}
|
||||
.live-badge.up .live-dot::after { background: rgba(16, 185, 129, 0.3); }
|
||||
.live-badge.down .live-dot::after { background: rgba(239, 68, 68, 0.3); }
|
||||
@keyframes pulse { 0% { transform: scale(1); opacity: 1; } 100% { transform: scale(2.5); opacity: 0; } }
|
||||
|
||||
.updated-text { font-size: 11px; color: var(--text-muted); }
|
||||
|
||||
/* ── Main ── */
|
||||
.main { padding: 28px 32px; max-width: 1440px; margin: 0 auto; }
|
||||
|
||||
/* ── Top Cards ── */
|
||||
.top-grid {
|
||||
display: grid;
|
||||
grid-template-columns: repeat(5, 1fr);
|
||||
gap: 16px;
|
||||
margin-bottom: 24px;
|
||||
}
|
||||
|
||||
.card {
|
||||
background: var(--bg-card);
|
||||
border: 1px solid var(--border);
|
||||
border-radius: var(--radius);
|
||||
padding: 20px;
|
||||
backdrop-filter: blur(12px);
|
||||
transition: all 0.3s ease;
|
||||
position: relative;
|
||||
overflow: hidden;
|
||||
}
|
||||
.card:hover { border-color: var(--border-hover); transform: translateY(-1px); }
|
||||
.card::before {
|
||||
content: '';
|
||||
position: absolute; top: 0; left: 0; right: 0; height: 2px;
|
||||
opacity: 0; transition: opacity 0.3s;
|
||||
}
|
||||
.card:hover::before { opacity: 1; }
|
||||
|
||||
.card.green::before { background: linear-gradient(90deg, var(--green), transparent); }
|
||||
.card.blue::before { background: linear-gradient(90deg, var(--blue), transparent); }
|
||||
.card.amber::before { background: linear-gradient(90deg, var(--amber), transparent); }
|
||||
.card.purple::before { background: linear-gradient(90deg, var(--purple), transparent); }
|
||||
.card.cyan::before { background: linear-gradient(90deg, var(--cyan), transparent); }
|
||||
|
||||
.card-icon {
|
||||
width: 36px; height: 36px; border-radius: 10px;
|
||||
display: flex; align-items: center; justify-content: center;
|
||||
font-size: 17px; margin-bottom: 14px;
|
||||
}
|
||||
.card.green .card-icon { background: var(--green-glow); }
|
||||
.card.blue .card-icon { background: var(--blue-glow); }
|
||||
.card.amber .card-icon { background: var(--amber-glow); }
|
||||
.card.purple .card-icon { background: var(--purple-glow); }
|
||||
.card.cyan .card-icon { background: var(--cyan-glow); }
|
||||
|
||||
.card-label {
|
||||
font-size: 11px; color: var(--text-muted); font-weight: 600;
|
||||
text-transform: uppercase; letter-spacing: 0.7px; margin-bottom: 6px;
|
||||
}
|
||||
.card-value {
|
||||
font-size: 26px; font-weight: 800; letter-spacing: -0.8px; line-height: 1;
|
||||
}
|
||||
.card.green .card-value { color: var(--green); }
|
||||
.card.blue .card-value { color: var(--blue); }
|
||||
.card.amber .card-value { color: var(--amber); }
|
||||
.card.purple .card-value { color: var(--purple); }
|
||||
.card.cyan .card-value { color: var(--cyan); }
|
||||
|
||||
.card-sub { font-size: 11px; color: var(--text-muted); margin-top: 6px; }
|
||||
.card-unit { font-size: 13px; font-weight: 400; }
|
||||
|
||||
/* ── Panels ── */
|
||||
.mid-grid {
|
||||
display: grid;
|
||||
grid-template-columns: 2fr 1fr;
|
||||
gap: 16px;
|
||||
margin-bottom: 24px;
|
||||
}
|
||||
|
||||
.panel {
|
||||
background: var(--bg-card);
|
||||
border: 1px solid var(--border);
|
||||
border-radius: var(--radius);
|
||||
backdrop-filter: blur(12px);
|
||||
overflow: hidden;
|
||||
}
|
||||
.panel-header {
|
||||
padding: 16px 20px;
|
||||
border-bottom: 1px solid var(--border);
|
||||
display: flex; align-items: center; justify-content: space-between;
|
||||
}
|
||||
.panel-title { font-size: 14px; font-weight: 600; display: flex; align-items: center; gap: 8px; }
|
||||
.panel-body { padding: 20px; }
|
||||
|
||||
/* ── Line Chart ── */
|
||||
.chart-area {
|
||||
width: 100%; height: 220px; position: relative;
|
||||
}
|
||||
.chart-area canvas { width: 100% !important; height: 100% !important; }
|
||||
.chart-empty {
|
||||
height: 100%; display: flex; align-items: center; justify-content: center;
|
||||
color: var(--text-muted); font-size: 13px;
|
||||
}
|
||||
|
||||
/* ── SSL Panel ── */
|
||||
.ssl-info { display: flex; flex-direction: column; gap: 16px; }
|
||||
|
||||
.ssl-main {
|
||||
text-align: center; padding: 20px;
|
||||
background: rgba(255, 255, 255, 0.02);
|
||||
border-radius: var(--radius-sm);
|
||||
border: 1px solid var(--border);
|
||||
}
|
||||
.ssl-days {
|
||||
font-size: 48px; font-weight: 900; line-height: 1;
|
||||
background: linear-gradient(135deg, var(--green), var(--cyan));
|
||||
-webkit-background-clip: text; -webkit-text-fill-color: transparent;
|
||||
background-clip: text;
|
||||
}
|
||||
.ssl-days.warning {
|
||||
background: linear-gradient(135deg, var(--amber), var(--red));
|
||||
-webkit-background-clip: text; background-clip: text;
|
||||
}
|
||||
.ssl-days-label { font-size: 12px; color: var(--text-muted); margin-top: 6px; text-transform: uppercase; letter-spacing: 0.5px; }
|
||||
|
||||
.ssl-detail {
|
||||
display: flex; justify-content: space-between;
|
||||
padding: 10px 0;
|
||||
border-bottom: 1px solid var(--border);
|
||||
font-size: 13px;
|
||||
}
|
||||
.ssl-detail:last-child { border-bottom: none; }
|
||||
.ssl-detail-label { color: var(--text-muted); }
|
||||
.ssl-detail-value { font-weight: 600; }
|
||||
|
||||
/* ── Progress Bar (Uptime) ── */
|
||||
.uptime-bar-container {
|
||||
background: rgba(255, 255, 255, 0.04);
|
||||
border-radius: 8px;
|
||||
height: 10px;
|
||||
overflow: hidden;
|
||||
margin-top: 10px;
|
||||
}
|
||||
.uptime-bar {
|
||||
height: 100%;
|
||||
border-radius: 8px;
|
||||
background: linear-gradient(90deg, var(--green), var(--cyan));
|
||||
transition: width 1s cubic-bezier(0.22, 1, 0.36, 1);
|
||||
}
|
||||
|
||||
/* ── Records Table ── */
|
||||
.records-panel { margin-bottom: 24px; }
|
||||
|
||||
.table-wrapper { overflow-x: auto; }
|
||||
|
||||
table {
|
||||
width: 100%; border-collapse: collapse;
|
||||
font-size: 12.5px;
|
||||
}
|
||||
th {
|
||||
text-align: left; padding: 10px 14px;
|
||||
color: var(--text-muted); font-weight: 600;
|
||||
text-transform: uppercase; letter-spacing: 0.5px;
|
||||
font-size: 11px; border-bottom: 1px solid var(--border);
|
||||
white-space: nowrap;
|
||||
}
|
||||
td {
|
||||
padding: 11px 14px;
|
||||
border-bottom: 1px solid rgba(255, 255, 255, 0.025);
|
||||
white-space: nowrap;
|
||||
}
|
||||
tr:hover td { background: rgba(255, 255, 255, 0.015); }
|
||||
|
||||
.badge {
|
||||
display: inline-flex; padding: 3px 10px; border-radius: 100px;
|
||||
font-size: 11px; font-weight: 600;
|
||||
}
|
||||
.badge.up { background: var(--green-glow); color: var(--green); }
|
||||
.badge.down { background: var(--red-glow); color: var(--red); }
|
||||
.badge.warn { background: var(--amber-glow); color: var(--amber); }
|
||||
|
||||
.no-data {
|
||||
text-align: center; padding: 40px;
|
||||
color: var(--text-muted); font-size: 13px;
|
||||
}
|
||||
|
||||
/* ── Responsive ── */
|
||||
@media (max-width: 1100px) {
|
||||
.top-grid { grid-template-columns: repeat(3, 1fr); }
|
||||
.mid-grid { grid-template-columns: 1fr; }
|
||||
}
|
||||
@media (max-width: 700px) {
|
||||
.top-grid { grid-template-columns: repeat(2, 1fr); }
|
||||
.main { padding: 16px; }
|
||||
.header { padding: 14px 16px; }
|
||||
.header-title span { display: none; }
|
||||
}
|
||||
@media (max-width: 480px) {
|
||||
.top-grid { grid-template-columns: 1fr; }
|
||||
}
|
||||
</style>
|
||||
</head>
|
||||
<body>
|
||||
<div class="app">
|
||||
|
||||
<!-- ═══ Header ═══ -->
|
||||
<header class="header">
|
||||
<div class="header-left">
|
||||
<div class="logo">M</div>
|
||||
<h1 class="header-title">mithal.space <span>Monitoring Dashboard</span></h1>
|
||||
</div>
|
||||
<div class="header-right">
|
||||
<div id="liveBadge" class="live-badge unknown">
|
||||
<div class="live-dot"></div>
|
||||
<span id="liveText">LOADING...</span>
|
||||
</div>
|
||||
<div class="updated-text">Updated: <span id="lastUpdated">—</span></div>
|
||||
</div>
|
||||
</header>
|
||||
|
||||
<!-- ═══ Main ═══ -->
|
||||
<main class="main">
|
||||
|
||||
<!-- ── Top Metric Cards ── -->
|
||||
<section class="top-grid">
|
||||
<div class="card green" id="uptimeCard">
|
||||
<div class="card-icon">🟢</div>
|
||||
<div class="card-label">Uptime (24h)</div>
|
||||
<div class="card-value" id="uptimeValue">—<span class="card-unit">%</span></div>
|
||||
<div class="uptime-bar-container"><div class="uptime-bar" id="uptimeBar" style="width: 0%"></div></div>
|
||||
<div class="card-sub" id="uptimeSub">0 / 0 checks passed</div>
|
||||
</div>
|
||||
|
||||
<div class="card blue">
|
||||
<div class="card-icon">⚡</div>
|
||||
<div class="card-label">Avg Latency</div>
|
||||
<div class="card-value" id="avgLatency">—<span class="card-unit"> ms</span></div>
|
||||
<div class="card-sub" id="latencySub">Last hour</div>
|
||||
</div>
|
||||
|
||||
<div class="card cyan">
|
||||
<div class="card-icon">🌐</div>
|
||||
<div class="card-label">DNS Resolution</div>
|
||||
<div class="card-value" id="dnsValue">—<span class="card-unit"> ms</span></div>
|
||||
<div class="card-sub" id="dnsSub">—</div>
|
||||
</div>
|
||||
|
||||
<div class="card amber">
|
||||
<div class="card-icon">🔒</div>
|
||||
<div class="card-label">SSL Certificate</div>
|
||||
<div class="card-value" id="sslDays">—<span class="card-unit"> days</span></div>
|
||||
<div class="card-sub" id="sslSub">—</div>
|
||||
</div>
|
||||
|
||||
<div class="card purple">
|
||||
<div class="card-icon">🔍</div>
|
||||
<div class="card-label">Search Response</div>
|
||||
<div class="card-value" id="searchValue">—<span class="card-unit"> ms</span></div>
|
||||
<div class="card-sub" id="searchSub">—</div>
|
||||
</div>
|
||||
</section>
|
||||
|
||||
<!-- ── Chart + SSL Details ── -->
|
||||
<section class="mid-grid">
|
||||
<div class="panel">
|
||||
<div class="panel-header">
|
||||
<div class="panel-title">📈 Response Time — Last Hour</div>
|
||||
</div>
|
||||
<div class="panel-body">
|
||||
<div class="chart-area" id="chartArea">
|
||||
<canvas id="latencyChart"></canvas>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<div class="panel">
|
||||
<div class="panel-header">
|
||||
<div class="panel-title">🔒 SSL Certificate Details</div>
|
||||
</div>
|
||||
<div class="panel-body">
|
||||
<div class="ssl-info">
|
||||
<div class="ssl-main">
|
||||
<div class="ssl-days" id="sslDaysBig">—</div>
|
||||
<div class="ssl-days-label">Days Until Expiration</div>
|
||||
</div>
|
||||
<div class="ssl-detail">
|
||||
<span class="ssl-detail-label">Status</span>
|
||||
<span class="ssl-detail-value" id="sslStatus">—</span>
|
||||
</div>
|
||||
<div class="ssl-detail">
|
||||
<span class="ssl-detail-label">Issuer</span>
|
||||
<span class="ssl-detail-value" id="sslIssuer">—</span>
|
||||
</div>
|
||||
<div class="ssl-detail">
|
||||
<span class="ssl-detail-label">Expires On</span>
|
||||
<span class="ssl-detail-value" id="sslExpires">—</span>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
</section>
|
||||
|
||||
<!-- ── Last 10 Records Table ── -->
|
||||
<section class="records-panel">
|
||||
<div class="panel">
|
||||
<div class="panel-header">
|
||||
<div class="panel-title">📋 Last 10 Test Results</div>
|
||||
</div>
|
||||
<div class="panel-body">
|
||||
<div class="table-wrapper">
|
||||
<table>
|
||||
<thead>
|
||||
<tr>
|
||||
<th>Time</th>
|
||||
<th>Status</th>
|
||||
<th>HTTP</th>
|
||||
<th>Latency</th>
|
||||
<th>DNS</th>
|
||||
<th>Search</th>
|
||||
<th>SSL</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody id="recordsBody">
|
||||
<tr><td colspan="7" class="no-data">Waiting for data from collector...</td></tr>
|
||||
</tbody>
|
||||
</table>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
</section>
|
||||
|
||||
</main>
|
||||
</div>
|
||||
|
||||
<!-- Chart.js for line chart -->
|
||||
<script src="https://cdn.jsdelivr.net/npm/chart.js@4.4.4/dist/chart.umd.min.js"></script>
|
||||
|
||||
<script>
|
||||
// ═══════════════════════════════════════
|
||||
// Config
|
||||
// ═══════════════════════════════════════
|
||||
const DATA_URL = '/api/monitoring/metrics';
|
||||
const POLL_INTERVAL = 10000; // 10 seconds
|
||||
let chart = null;
|
||||
|
||||
// ═══════════════════════════════════════
|
||||
// Fetch & Update
|
||||
// ═══════════════════════════════════════
|
||||
async function fetchData() {
|
||||
try {
|
||||
const res = await fetch(DATA_URL);
|
||||
if (!res.ok) throw new Error(`HTTP ${res.status}`);
|
||||
const data = await res.json();
|
||||
updateDashboard(data);
|
||||
} catch (err) {
|
||||
setStatus('unknown', 'NO DATA');
|
||||
console.error('Fetch error:', err);
|
||||
}
|
||||
}
|
||||
|
||||
function updateDashboard(data) {
|
||||
const records = data.records || [];
|
||||
if (records.length === 0) {
|
||||
setStatus('unknown', 'NO DATA');
|
||||
return;
|
||||
}
|
||||
|
||||
const latest = records[records.length - 1];
|
||||
document.getElementById('lastUpdated').textContent = formatTime(latest.timestamp);
|
||||
|
||||
// ── Live status ──
|
||||
if (latest.uptime.is_up) {
|
||||
setStatus('up', 'ONLINE');
|
||||
} else {
|
||||
setStatus('down', 'OFFLINE');
|
||||
}
|
||||
|
||||
// ── Uptime (24h) ──
|
||||
const upCount = records.filter(r => r.uptime.is_up).length;
|
||||
const total = records.length;
|
||||
const uptimePct = total > 0 ? ((upCount / total) * 100).toFixed(2) : 0;
|
||||
document.getElementById('uptimeValue').innerHTML = `${uptimePct}<span class="card-unit">%</span>`;
|
||||
document.getElementById('uptimeBar').style.width = `${uptimePct}%`;
|
||||
document.getElementById('uptimeSub').textContent = `${upCount} / ${total} checks passed`;
|
||||
|
||||
// ── Avg Latency (last hour = last 60 records max) ──
|
||||
const lastHour = records.slice(-60);
|
||||
const latencies = lastHour.map(r => r.latency.latency_ms).filter(v => v !== null);
|
||||
const avgLat = latencies.length > 0 ? Math.round(latencies.reduce((a, b) => a + b, 0) / latencies.length) : 0;
|
||||
document.getElementById('avgLatency').innerHTML = `${avgLat}<span class="card-unit"> ms</span>`;
|
||||
document.getElementById('latencySub').textContent = `${latencies.length} samples`;
|
||||
|
||||
// ── DNS ──
|
||||
const dnsTime = latest.dns.dns_time_ms;
|
||||
document.getElementById('dnsValue').innerHTML = dnsTime !== null
|
||||
? `${dnsTime}<span class="card-unit"> ms</span>`
|
||||
: `—<span class="card-unit"> ms</span>`;
|
||||
document.getElementById('dnsSub').textContent = latest.dns.resolved_ip || '—';
|
||||
|
||||
// ── SSL ──
|
||||
const sslDays = latest.ssl.days_remaining;
|
||||
document.getElementById('sslDays').innerHTML = sslDays !== null
|
||||
? `${sslDays}<span class="card-unit"> days</span>`
|
||||
: `—<span class="card-unit"></span>`;
|
||||
document.getElementById('sslSub').textContent = latest.ssl.valid ? 'Valid' : 'Invalid / Error';
|
||||
|
||||
// SSL Detail panel
|
||||
const sslBigEl = document.getElementById('sslDaysBig');
|
||||
sslBigEl.textContent = sslDays !== null ? sslDays : '—';
|
||||
sslBigEl.className = (sslDays !== null && sslDays < 30) ? 'ssl-days warning' : 'ssl-days';
|
||||
document.getElementById('sslStatus').textContent = latest.ssl.valid ? '✅ Valid' : '❌ Invalid';
|
||||
document.getElementById('sslIssuer').textContent = latest.ssl.issuer || '—';
|
||||
document.getElementById('sslExpires').textContent = latest.ssl.expires || '—';
|
||||
|
||||
// ── Search ──
|
||||
const searchTime = latest.search.search_time_ms;
|
||||
document.getElementById('searchValue').innerHTML = searchTime !== null
|
||||
? `${searchTime}<span class="card-unit"> ms</span>`
|
||||
: `—<span class="card-unit"> ms</span>`;
|
||||
document.getElementById('searchSub').textContent = latest.search.content_length
|
||||
? `${(latest.search.content_length / 1024).toFixed(1)} KB response`
|
||||
: '—';
|
||||
|
||||
// ── Line Chart (last hour) ──
|
||||
renderChart(lastHour);
|
||||
|
||||
// ── Last 10 Records Table ──
|
||||
renderTable(records.slice(-10).reverse());
|
||||
}
|
||||
|
||||
// ═══════════════════════════════════════
|
||||
// Status Badge
|
||||
// ═══════════════════════════════════════
|
||||
function setStatus(state, text) {
|
||||
const badge = document.getElementById('liveBadge');
|
||||
const label = document.getElementById('liveText');
|
||||
badge.className = `live-badge ${state}`;
|
||||
label.textContent = text;
|
||||
}
|
||||
|
||||
// ═══════════════════════════════════════
|
||||
// Line Chart
|
||||
// ═══════════════════════════════════════
|
||||
function renderChart(records) {
|
||||
const labels = records.map(r => formatTime(r.timestamp));
|
||||
const latencies = records.map(r => r.latency.latency_ms || 0);
|
||||
|
||||
if (chart) {
|
||||
chart.data.labels = labels;
|
||||
chart.data.datasets[0].data = latencies;
|
||||
chart.update('none');
|
||||
return;
|
||||
}
|
||||
|
||||
const ctx = document.getElementById('latencyChart').getContext('2d');
|
||||
const gradient = ctx.createLinearGradient(0, 0, 0, 220);
|
||||
gradient.addColorStop(0, 'rgba(99, 102, 241, 0.25)');
|
||||
gradient.addColorStop(1, 'rgba(99, 102, 241, 0)');
|
||||
|
||||
chart = new Chart(ctx, {
|
||||
type: 'line',
|
||||
data: {
|
||||
labels: labels,
|
||||
datasets: [{
|
||||
label: 'Response Time (ms)',
|
||||
data: latencies,
|
||||
borderColor: '#6366f1',
|
||||
backgroundColor: gradient,
|
||||
borderWidth: 2,
|
||||
tension: 0.35,
|
||||
fill: true,
|
||||
pointRadius: 2,
|
||||
pointHoverRadius: 5,
|
||||
pointBackgroundColor: '#6366f1',
|
||||
pointHoverBackgroundColor: '#a855f7',
|
||||
}]
|
||||
},
|
||||
options: {
|
||||
responsive: true,
|
||||
maintainAspectRatio: false,
|
||||
interaction: { mode: 'index', intersect: false },
|
||||
plugins: {
|
||||
legend: { display: false },
|
||||
tooltip: {
|
||||
backgroundColor: 'rgba(12, 17, 32, 0.95)',
|
||||
borderColor: 'rgba(99, 102, 241, 0.3)',
|
||||
borderWidth: 1,
|
||||
titleColor: '#e8edf5',
|
||||
bodyColor: '#8892a8',
|
||||
padding: 12,
|
||||
cornerRadius: 8,
|
||||
callbacks: {
|
||||
label: (ctx) => `${ctx.parsed.y} ms`
|
||||
}
|
||||
}
|
||||
},
|
||||
scales: {
|
||||
x: {
|
||||
ticks: { color: '#525d75', font: { size: 10 }, maxTicksLimit: 10, maxRotation: 0 },
|
||||
grid: { color: 'rgba(255, 255, 255, 0.03)' },
|
||||
border: { color: 'transparent' }
|
||||
},
|
||||
y: {
|
||||
ticks: { color: '#525d75', font: { size: 10 }, callback: v => v + 'ms' },
|
||||
grid: { color: 'rgba(255, 255, 255, 0.03)' },
|
||||
border: { color: 'transparent' },
|
||||
beginAtZero: true
|
||||
}
|
||||
}
|
||||
}
|
||||
});
|
||||
}
|
||||
|
||||
// ═══════════════════════════════════════
|
||||
// Records Table
|
||||
// ═══════════════════════════════════════
|
||||
function renderTable(records) {
|
||||
const tbody = document.getElementById('recordsBody');
|
||||
|
||||
if (records.length === 0) {
|
||||
tbody.innerHTML = '<tr><td colspan="7" class="no-data">No records yet — start the collector</td></tr>';
|
||||
return;
|
||||
}
|
||||
|
||||
tbody.innerHTML = records.map(r => {
|
||||
const isUp = r.uptime.is_up;
|
||||
const statusBadge = isUp
|
||||
? '<span class="badge up">UP</span>'
|
||||
: '<span class="badge down">DOWN</span>';
|
||||
const httpCode = r.latency.status_code || '—';
|
||||
const lat = r.latency.latency_ms !== null ? `${r.latency.latency_ms}ms` : '—';
|
||||
const dns = r.dns.dns_time_ms !== null ? `${r.dns.dns_time_ms}ms` : '—';
|
||||
const search = r.search.search_time_ms !== null ? `${r.search.search_time_ms}ms` : '—';
|
||||
const sslBadge = r.ssl.valid
|
||||
? `<span class="badge up">${r.ssl.days_remaining}d</span>`
|
||||
: '<span class="badge down">ERR</span>';
|
||||
|
||||
return `<tr>
|
||||
<td>${formatTime(r.timestamp)}</td>
|
||||
<td>${statusBadge}</td>
|
||||
<td>${httpCode}</td>
|
||||
<td>${lat}</td>
|
||||
<td>${dns}</td>
|
||||
<td>${search}</td>
|
||||
<td>${sslBadge}</td>
|
||||
</tr>`;
|
||||
}).join('');
|
||||
}
|
||||
|
||||
// ═══════════════════════════════════════
|
||||
// Helpers
|
||||
// ═══════════════════════════════════════
|
||||
function formatTime(isoStr) {
|
||||
try {
|
||||
const d = new Date(isoStr);
|
||||
return d.toLocaleTimeString('en-GB', { hour: '2-digit', minute: '2-digit', second: '2-digit' });
|
||||
} catch {
|
||||
return isoStr;
|
||||
}
|
||||
}
|
||||
|
||||
// ═══════════════════════════════════════
|
||||
// Start
|
||||
// ═══════════════════════════════════════
|
||||
fetchData();
|
||||
setInterval(fetchData, POLL_INTERVAL);
|
||||
</script>
|
||||
</body>
|
||||
</html>
|
||||
@@ -0,0 +1,275 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
Monitoring Collector for mithal.space
|
||||
Collects metrics every 60 seconds: Latency, Uptime, SSL, DNS, Search Response.
|
||||
Stores results in data/mithal-metrics.json.
|
||||
"""
|
||||
|
||||
import urllib.request
|
||||
import urllib.error
|
||||
import ssl
|
||||
import socket
|
||||
import json
|
||||
import time
|
||||
import datetime
|
||||
import os
|
||||
import sys
|
||||
|
||||
# ─────────────────────────────────────────────
|
||||
# Configuration
|
||||
# ─────────────────────────────────────────────
|
||||
TARGET_URL = "https://mithal.space"
|
||||
SEARCH_URL = "https://mithal.space" # Search/query endpoint (adjust if different)
|
||||
CHECK_INTERVAL = 60 # seconds (1 minute)
|
||||
DATA_DIR = os.path.join(os.path.dirname(os.path.abspath(__file__)), "data")
|
||||
DATA_FILE = os.path.join(DATA_DIR, "mithal-metrics.json")
|
||||
MAX_RECORDS = 1440 # Keep 24 hours of data (1 per minute)
|
||||
TIMEOUT = 10 # seconds
|
||||
|
||||
# ANSI colors
|
||||
GREEN = "\033[92m"
|
||||
RED = "\033[91m"
|
||||
YELLOW = "\033[93m"
|
||||
CYAN = "\033[96m"
|
||||
BOLD = "\033[1m"
|
||||
RESET = "\033[0m"
|
||||
|
||||
|
||||
def ensure_data_dir():
|
||||
"""Create data directory if it doesn't exist."""
|
||||
os.makedirs(DATA_DIR, exist_ok=True)
|
||||
if not os.path.exists(DATA_FILE):
|
||||
with open(DATA_FILE, "w", encoding="utf-8") as f:
|
||||
json.dump({"records": []}, f)
|
||||
|
||||
|
||||
def load_data():
|
||||
"""Load existing metrics data."""
|
||||
try:
|
||||
with open(DATA_FILE, "r", encoding="utf-8") as f:
|
||||
return json.load(f)
|
||||
except (json.JSONDecodeError, FileNotFoundError):
|
||||
return {"records": []}
|
||||
|
||||
|
||||
def save_data(data):
|
||||
"""Save metrics data, keeping only the last MAX_RECORDS entries."""
|
||||
data["records"] = data["records"][-MAX_RECORDS:]
|
||||
with open(DATA_FILE, "w", encoding="utf-8") as f:
|
||||
json.dump(data, f, indent=2, ensure_ascii=False)
|
||||
|
||||
|
||||
# ─────────────────────────────────────────────
|
||||
# Metric Collectors
|
||||
# ─────────────────────────────────────────────
|
||||
|
||||
def check_latency(url):
|
||||
"""Measure HTTP response time (latency) in milliseconds."""
|
||||
try:
|
||||
start = time.time()
|
||||
req = urllib.request.Request(url, method="GET")
|
||||
req.add_header("User-Agent", "GhaymaMonitor/1.0")
|
||||
response = urllib.request.urlopen(req, timeout=TIMEOUT)
|
||||
latency_ms = round((time.time() - start) * 1000)
|
||||
status_code = response.status
|
||||
response.close()
|
||||
return {
|
||||
"latency_ms": latency_ms,
|
||||
"status_code": status_code,
|
||||
"is_up": 200 <= status_code < 400,
|
||||
"error": None
|
||||
}
|
||||
except urllib.error.HTTPError as e:
|
||||
latency_ms = round((time.time() - start) * 1000)
|
||||
return {
|
||||
"latency_ms": latency_ms,
|
||||
"status_code": e.code,
|
||||
"is_up": False,
|
||||
"error": f"HTTP {e.code}: {e.reason}"
|
||||
}
|
||||
except Exception as e:
|
||||
return {
|
||||
"latency_ms": None,
|
||||
"status_code": None,
|
||||
"is_up": False,
|
||||
"error": str(e)
|
||||
}
|
||||
|
||||
|
||||
def check_ssl(hostname):
|
||||
"""Check SSL certificate status and expiration date."""
|
||||
try:
|
||||
context = ssl.create_default_context()
|
||||
with socket.create_connection((hostname, 443), timeout=TIMEOUT) as sock:
|
||||
with context.wrap_socket(sock, server_hostname=hostname) as ssock:
|
||||
cert = ssock.getpeercert()
|
||||
|
||||
# Parse expiration date
|
||||
expire_str = cert.get("notAfter", "")
|
||||
expire_date = datetime.datetime.strptime(
|
||||
expire_str, "%b %d %H:%M:%S %Y %Z"
|
||||
)
|
||||
now = datetime.datetime.utcnow()
|
||||
days_remaining = (expire_date - now).days
|
||||
|
||||
# Get issuer
|
||||
issuer_parts = dict(x[0] for x in cert.get("issuer", []))
|
||||
issuer = issuer_parts.get("organizationName", "Unknown")
|
||||
|
||||
return {
|
||||
"valid": True,
|
||||
"issuer": issuer,
|
||||
"expires": expire_date.strftime("%Y-%m-%d %H:%M:%S UTC"),
|
||||
"days_remaining": days_remaining,
|
||||
"error": None
|
||||
}
|
||||
except Exception as e:
|
||||
return {
|
||||
"valid": False,
|
||||
"issuer": None,
|
||||
"expires": None,
|
||||
"days_remaining": None,
|
||||
"error": str(e)
|
||||
}
|
||||
|
||||
|
||||
def check_dns(hostname):
|
||||
"""Measure DNS resolution time in milliseconds."""
|
||||
try:
|
||||
start = time.time()
|
||||
ip_address = socket.gethostbyname(hostname)
|
||||
dns_time_ms = round((time.time() - start) * 1000)
|
||||
return {
|
||||
"dns_time_ms": dns_time_ms,
|
||||
"resolved_ip": ip_address,
|
||||
"error": None
|
||||
}
|
||||
except Exception as e:
|
||||
return {
|
||||
"dns_time_ms": None,
|
||||
"resolved_ip": None,
|
||||
"error": str(e)
|
||||
}
|
||||
|
||||
|
||||
def check_search_response(url):
|
||||
"""Send a search/query request and measure response time."""
|
||||
try:
|
||||
search_url = url.rstrip("/") + "/"
|
||||
start = time.time()
|
||||
req = urllib.request.Request(search_url, method="GET")
|
||||
req.add_header("User-Agent", "GhaymaMonitor/1.0")
|
||||
response = urllib.request.urlopen(req, timeout=TIMEOUT)
|
||||
search_time_ms = round((time.time() - start) * 1000)
|
||||
content_length = len(response.read())
|
||||
response.close()
|
||||
return {
|
||||
"search_time_ms": search_time_ms,
|
||||
"content_length": content_length,
|
||||
"status_code": response.status,
|
||||
"error": None
|
||||
}
|
||||
except Exception as e:
|
||||
return {
|
||||
"search_time_ms": None,
|
||||
"content_length": None,
|
||||
"status_code": None,
|
||||
"error": str(e)
|
||||
}
|
||||
|
||||
|
||||
# ─────────────────────────────────────────────
|
||||
# Main Collector
|
||||
# ─────────────────────────────────────────────
|
||||
|
||||
def collect_metrics():
|
||||
"""Run all metric checks and return a combined record."""
|
||||
timestamp = datetime.datetime.utcnow().strftime("%Y-%m-%dT%H:%M:%SZ")
|
||||
hostname = TARGET_URL.replace("https://", "").replace("http://", "").split("/")[0]
|
||||
|
||||
print(f"\n{CYAN}[{timestamp}] Collecting metrics for {TARGET_URL}...{RESET}")
|
||||
|
||||
# 1. DNS Resolution
|
||||
dns_result = check_dns(hostname)
|
||||
status = f"{GREEN}✅ {dns_result['dns_time_ms']}ms{RESET}" if not dns_result["error"] else f"{RED}❌ {dns_result['error']}{RESET}"
|
||||
print(f" DNS Resolution: {status}")
|
||||
|
||||
# 2. HTTP Latency & Uptime
|
||||
latency_result = check_latency(TARGET_URL)
|
||||
if latency_result["is_up"]:
|
||||
status = f"{GREEN}✅ {latency_result['status_code']} in {latency_result['latency_ms']}ms{RESET}"
|
||||
else:
|
||||
status = f"{RED}❌ {latency_result.get('error', 'Down')}{RESET}"
|
||||
print(f" HTTP Latency: {status}")
|
||||
|
||||
# 3. SSL Certificate
|
||||
ssl_result = check_ssl(hostname)
|
||||
if ssl_result["valid"]:
|
||||
days = ssl_result["days_remaining"]
|
||||
color = GREEN if days > 30 else (YELLOW if days > 7 else RED)
|
||||
status = f"{color}✅ Valid, {days} days remaining{RESET}"
|
||||
else:
|
||||
status = f"{RED}❌ {ssl_result.get('error', 'Invalid')}{RESET}"
|
||||
print(f" SSL Status: {status}")
|
||||
|
||||
# 4. Search Response
|
||||
search_result = check_search_response(SEARCH_URL)
|
||||
if search_result["search_time_ms"]:
|
||||
status = f"{GREEN}✅ {search_result['search_time_ms']}ms ({search_result['content_length']} bytes){RESET}"
|
||||
else:
|
||||
status = f"{RED}❌ {search_result.get('error', 'Failed')}{RESET}"
|
||||
print(f" Search Response: {status}")
|
||||
|
||||
# Combine into a single record
|
||||
record = {
|
||||
"timestamp": timestamp,
|
||||
"latency": latency_result,
|
||||
"uptime": {
|
||||
"is_up": latency_result["is_up"],
|
||||
"status_code": latency_result["status_code"]
|
||||
},
|
||||
"ssl": ssl_result,
|
||||
"dns": dns_result,
|
||||
"search": search_result
|
||||
}
|
||||
|
||||
return record
|
||||
|
||||
|
||||
def main():
|
||||
ensure_data_dir()
|
||||
|
||||
print(f"{BOLD}{CYAN}")
|
||||
print("=" * 58)
|
||||
print(" mithal.space — Monitoring Collector")
|
||||
print(f" Target: {TARGET_URL}")
|
||||
print(f" Interval: {CHECK_INTERVAL}s (every minute)")
|
||||
print(f" Storage: {DATA_FILE}")
|
||||
print("=" * 58)
|
||||
print(f"{RESET}")
|
||||
|
||||
try:
|
||||
while True:
|
||||
record = collect_metrics()
|
||||
|
||||
# Load, append, save
|
||||
data = load_data()
|
||||
data["records"].append(record)
|
||||
save_data(data)
|
||||
|
||||
up_count = sum(1 for r in data["records"] if r["uptime"]["is_up"])
|
||||
total = len(data["records"])
|
||||
uptime_pct = round((up_count / total) * 100, 2) if total > 0 else 0
|
||||
|
||||
print(f" {BOLD}Uptime (session): {uptime_pct}% ({up_count}/{total} checks){RESET}")
|
||||
print(f" {CYAN}Next check in {CHECK_INTERVAL}s...{RESET}")
|
||||
|
||||
time.sleep(CHECK_INTERVAL)
|
||||
|
||||
except KeyboardInterrupt:
|
||||
print(f"\n{YELLOW}Collector stopped.{RESET}")
|
||||
sys.exit(0)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
Reference in New Issue
Block a user