الملفات

649 أسطر
33 KiB
Markdown
خام الرابط الدائم اللوم التاريخ

هذا الملف يحتوي على أحرف Unicode غامضة

هذا الملف يحتوي على أحرف Unicode قد تُخلط مع أحرف أخرى. إذا كنت تعتقد أن هذا مقصود، يمكنك تجاهل هذا التحذير بأمان. استخدم زر الهروب للكشف عنها.

# Scalability & Load Distribution — Architecture Report
**Project:** Ghayma REST API
**Platform:** ghaymah.systems (Ghayma Cloud)
**Target Load:** 15,000 requests/second
**Date:** July 26, 2026
---
## Table of Contents
1. [Architecture Diagram — 15,000 req/s](#1-architecture-diagram--15000-reqs)
2. [Container Capacity Calculation](#2-container-capacity-calculation)
3. [Cold Start Strategy for New Containers](#3-cold-start-strategy-for-new-containers)
4. [Ghayma Block Storage for Stateful Workloads](#4-ghayma-block-storage-for-stateful-workloads)
---
## 1. Architecture Diagram — 15,000 req/s
### High-Level System Architecture
```
15,000 req/s
┌───────────────────────┐
│ DNS / CDN │
│ (dns.ghaymah.systems)│
└───────────┬───────────┘
┌───────────────────────┐
│ LOAD BALANCER │
│ (Layer 7 - HTTPS) │
│ │
│ • SSL Termination │
│ • Health Checks │
│ • Round Robin / │
│ Least Connections │
│ • Rate Limiting │
└───────────┬───────────┘
┌────────────────────┼────────────────────┐
│ │ │
▼ ▼ ▼
┌─────────────┐ ┌─────────────┐ ┌─────────────┐
│ Zone A │ │ Zone B │ │ Zone C │
│ (Rack/AZ 1) │ │ (Rack/AZ 2) │ │ (Rack/AZ 3) │
└──────┬──────┘ └──────┬──────┘ └──────┬──────┘
│ │ │
┌──────┴──────┐ ┌──────┴──────┐ ┌──────┴──────┐
│ ┌─────────┐ │ │ ┌─────────┐ │ │ ┌─────────┐ │
│ │Container│ │ │ │Container│ │ │ │Container│ │
│ │ 1-15 │ │ │ │ 16-30 │ │ │ │ 31-44 │ │
│ │ (15x) │ │ │ │ (15x) │ │ │ │ (14x) │ │
│ └─────────┘ │ │ └─────────┘ │ │ └─────────┘ │
└──────┬──────┘ └──────┬──────┘ └──────┬──────┘
│ │ │
└───────────────────┼───────────────────┘
┌─────────────────┼─────────────────┐
│ │ │
▼ ▼ ▼
┌──────────────┐ ┌──────────────┐ ┌──────────────┐
│ Database │ │ Redis │ │ Block Storage│
│ (Primary + │ │ (Session & │ │ (Persistent │
│ Replicas) │ │ Cache) │ │ Volumes) │
└──────────────┘ └──────────────┘ └──────────────┘
```
### Detailed Component Architecture
```
┌──────────────────────────────────────────────────────────────────────────────────────┐
│ │
│ GHAYMA CLOUD PLATFORM │
│ │
│ ┌────────────────────────────────────────────────────────────────────────────────┐ │
│ │ NETWORKING LAYER │ │
│ │ │ │
│ │ Internet ──▶ [DNS] ──▶ [CDN/WAF] ──▶ [Load Balancer (L7)] │ │
│ │ │ │ │
│ │ ┌────────────────────┤ │ │
│ │ │ Health Checks │ │ │
│ │ │ GET /health (10s) │ │ │
│ │ │ Threshold: 3 fails │ │ │
│ │ └────────────────────┘ │ │
│ └────────────────────────────────┬───────────────────────────────────────────────┘ │
│ │ │
│ ┌────────────────────────────────┼───────────────────────────────────────────────┐ │
│ │ COMPUTE LAYER (44 Containers) │ │
│ │ │ │
│ │ ┌─────────┐ ┌─────────┐ ┌─────────┐ ┌─────────┐ ┌─────────┐ │ │
│ │ │ C-01 │ │ C-02 │ │ C-03 │ │ C-04 │ ... │ C-44 │ │ │
│ │ │ 500r/s │ │ 500r/s │ │ 500r/s │ │ 500r/s │ │ 500r/s │ │ │
│ │ │ 512MB │ │ 512MB │ │ 512MB │ │ 512MB │ │ 512MB │ │ │
│ │ │ 0.5 CPU │ │ 0.5 CPU │ │ 0.5 CPU │ │ 0.5 CPU │ │ 0.5 CPU │ │ │
│ │ └────┬────┘ └────┬────┘ └────┬────┘ └────┬────┘ └────┬────┘ │ │
│ │ └────────────┴────────────┴────────────┴──── ... ───────┘ │ │
│ │ │ │ │
│ │ ┌──────────────────────────────────────────────────────────────────┐ │ │
│ │ │ AUTO-SCALER (HPA) │ │ │
│ │ │ Min: 44 | Max: 66 | Scale at: CPU>75% or Mem>70% │ │ │
│ │ └──────────────────────────────────────────────────────────────────┘ │ │
│ └────────────────────────────────┬───────────────────────────────────────────────┘ │
│ │ │
│ ┌────────────────────────────────┼───────────────────────────────────────────────┐ │
│ │ DATA LAYER │ │
│ │ │ │
│ │ ┌──────────────────┐ ┌─────────────┐ ┌──────────────────────────────┐ │ │
│ │ │ PostgreSQL DB │ │ Redis │ │ Ghayma Block Storage │ │ │
│ │ │ Primary + 2 │ │ Cluster │ │ (Persistent Volumes) │ │ │
│ │ │ Read Replicas │ │ (Cache + │ │ • Database data (100GB) │ │ │
│ │ │ │ │ Sessions) │ │ • Logs & uploads (50GB) │ │ │
│ │ └──────────────────┘ └─────────────┘ └──────────────────────────────┘ │ │
│ └────────────────────────────────────────────────────────────────────────────────┘ │
│ │
│ ┌────────────────────────────────────────────────────────────────────────────────┐ │
│ │ OBSERVABILITY LAYER │ │
│ │ │ │
│ │ [Prometheus] ──▶ [Grafana Dashboards] ──▶ [AlertManager] ──▶ [PagerDuty] │ │
│ │ [Container Logs] ──▶ [Centralized Logging] ──▶ [Log Analytics] │ │
│ └────────────────────────────────────────────────────────────────────────────────┘ │
│ │
└──────────────────────────────────────────────────────────────────────────────────────┘
```
### Load Balancer Distribution Strategy
```
15,000 req/s incoming
┌────────┴────────┐
│ LOAD BALANCER │
│ │
│ Algorithm: │
│ Least │
│ Connections │
└────────┬────────┘
┌────────────────┼────────────────┐
│ │ │
Zone A (33%) Zone B (33%) Zone C (34%)
~5,000 r/s ~5,000 r/s ~5,000 r/s
15 containers 15 containers 14 containers
~333 r/s each ~333 r/s each ~357 r/s each
```
> Each container handles **~341 req/s on average** — well within the 500 req/s capacity, leaving the 30% safety margin intact.
---
## 2. Container Capacity Calculation
### Given Parameters
| Parameter | Value |
| :--- | :--- |
| **Total traffic** | 15,000 req/s |
| **Capacity per container** | 500 req/s |
| **Safety margin** | 30% |
### Step-by-Step Calculation
#### Step 1: Calculate Effective Capacity per Container
The 30% margin means each container should only be utilized at 70% of its max capacity:
```
Effective capacity = Max capacity × (1 - Safety margin)
Effective capacity = 500 req/s × (1 - 0.30)
Effective capacity = 500 × 0.70
Effective capacity = 350 req/s per container
```
#### Step 2: Calculate Minimum Containers Required
```
Minimum containers = Total traffic ÷ Effective capacity
Minimum containers = 15,000 ÷ 350
Minimum containers = 42.86
Minimum containers = 43 containers (rounded up)
```
#### Step 3: Add Availability Buffer (+1)
To handle single-container failures without breaching capacity:
```
Recommended containers = 43 + 1 (failure tolerance)
Recommended containers = 44 containers
```
### Summary Table
| Metric | Value | Formula |
| :--- | :--- | :--- |
| **Total load** | 15,000 req/s | Given |
| **Max capacity/container** | 500 req/s | Given |
| **Safety margin** | 30% | Given |
| **Effective capacity/container** | 350 req/s | 500 × 0.70 |
| **Minimum containers** | 43 | ⌈15,000 ÷ 350⌉ |
| **Recommended (with buffer)** | **44** | 43 + 1 |
| **Actual utilization per container** | ~341 req/s | 15,000 ÷ 44 |
| **Utilization percentage** | 68.2% | 341 ÷ 500 |
| **Remaining headroom** | 31.8% | ✅ Exceeds 30% margin |
### Capacity Planning Visualization
```
Container Capacity: 500 req/s
├─────────────────────────────────────────────────┤
0 500
Actual load per container: ~341 req/s (68.2%)
├──────────────────────────────────┤
0 341
Safety margin zone (30%):
├──────────────────────────────────┤░░░░░░░░░░░░░░┤
0 350 500
Threshold
(don't exceed)
```
### Auto-Scaling Tiers
| Traffic Level | Containers Needed | Status |
| :--- | :--- | :--- |
| 5,000 req/s (low) | 15 | Minimum viable |
| 10,000 req/s (moderate) | 30 | Normal scaling |
| **15,000 req/s (target)** | **44** | **Design target** |
| 20,000 req/s (spike) | 58 | Burst capacity |
| 25,000 req/s (peak) | 72 | Maximum scaling |
### Resource Allocation per Container
| Resource | Value |
| :--- | :--- |
| Memory Request | 256 Mi |
| Memory Limit | 512 Mi |
| CPU Request | 250m (0.25 vCPU) |
| CPU Limit | 500m (0.5 vCPU) |
### Total Cluster Resources (44 containers)
| Resource | Per Container | × 44 Containers | Total |
| :--- | :--- | :--- | :--- |
| **Memory** | 512 Mi | × 44 | **22 GB** |
| **CPU** | 0.5 vCPU | × 44 | **22 vCPUs** |
| **Throughput** | 350 req/s (effective) | × 44 | **15,400 req/s** |
---
## 3. Cold Start Strategy for New Containers
### What is a Cold Start?
A **cold start** occurs when a new container instance is created and must initialize before serving traffic. During this period:
- The container image is pulled from the registry
- The Node.js runtime starts
- Dependencies are loaded
- The application connects to databases/caches
- The first requests experience **higher latency**
```
Cold Start Timeline (without optimization)
──────────────────────────────────────────────────────
│ Image Pull │ Runtime Init │ App Boot │ DB Connect │ Ready │
0s 3s 4s 5s 7s 8s
First request served
```
### Cold Start Strategy — 5 Layers
#### Layer 1: Pre-Warmed Container Pool
Maintain a pool of **idle but ready** containers that can immediately accept traffic:
```
Active Containers (44) Pre-Warmed Pool (4-6)
┌────┬────┬────┬─ ... ─┬────┐ ┌────┬────┬────┬────┐
│ C1 │ C2 │ C3 │ │C44 │ │ W1 │ W2 │ W3 │ W4 │
│BUSY│BUSY│BUSY│ │BUSY│ │IDLE│IDLE│IDLE│IDLE│
└────┴────┴────┴─ ... ─┴────┘ └────┴────┴────┴────┘
Traffic spike detected
W1, W2 immediately start
serving (0s cold start)
```
**Implementation:**
```yaml
# HPA with pre-warmed minimum
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
spec:
minReplicas: 48 # 44 active + 4 pre-warmed
maxReplicas: 72
```
#### Layer 2: Optimized Docker Image (Fast Pulls)
Reduce image pull time by using lightweight images:
```dockerfile
# ✅ GOOD: Alpine-based image (~150MB, pulls in ~2s)
FROM node:20-alpine
# ❌ BAD: Full Debian image (~1GB, pulls in ~12s)
FROM node:20
```
| Image Base | Size | Pull Time | Cold Start Impact |
| :--- | :--- | :--- | :--- |
| `node:20` (Debian) | ~1 GB | ~12s | Very slow |
| `node:20-slim` | ~250 MB | ~5s | Moderate |
| `node:20-alpine` | ~150 MB | ~2s | ✅ **Fast** |
#### Layer 3: Application-Level Readiness
Implement a **readiness probe** that only marks the container as "ready" after it has fully initialized:
```javascript
// In src/index.js
let isReady = false;
// Readiness check - only returns 200 when app is fully initialized
app.get('/ready', (req, res) => {
if (isReady) {
res.status(200).json({ ready: true });
} else {
res.status(503).json({ ready: false, message: 'Warming up...' });
}
});
// App startup sequence
async function initialize() {
// 1. Connect to database
await connectDatabase();
// 2. Connect to Redis cache
await connectRedis();
// 3. Pre-load frequently accessed data into memory
await preloadCache();
// 4. Mark as ready ONLY after everything is initialized
isReady = true;
console.log('✅ Application fully initialized and ready');
}
app.listen(PORT, async () => {
console.log(`Server starting on port ${PORT}...`);
await initialize();
});
```
```yaml
# Kubernetes readiness probe
readinessProbe:
httpGet:
path: /ready # Not /health — separate from liveness
port: 3000
initialDelaySeconds: 3
periodSeconds: 5
failureThreshold: 3 # Must pass 3 checks before receiving traffic
```
> The load balancer **will not route traffic** to the container until `/ready` returns HTTP 200.
#### Layer 4: Proactive Scaling (Anticipate Demand)
Scale **before** the spike hits, not after:
```yaml
# CronJob-based pre-scaling for known traffic patterns
apiVersion: batch/v1
kind: CronJob
metadata:
name: pre-scale-morning
spec:
# Scale up at 8:00 AM before morning traffic rush
schedule: "0 8 * * 1-5"
jobTemplate:
spec:
template:
spec:
containers:
- name: scaler
image: bitnami/kubectl:latest
command:
- kubectl
- scale
- deployment/ghayma-api
- --replicas=50
```
#### Layer 5: Connection Pooling & Warm-Up Requests
After a container starts, send **internal warm-up requests** to pre-establish connections:
```javascript
async function warmUp() {
console.log('🔥 Running warm-up sequence...');
// Pre-establish database connection pool
await db.query('SELECT 1');
// Prime V8 JIT compiler with typical request patterns
const warmUpPaths = ['/health', '/api/v1/items'];
for (const path of warmUpPaths) {
await fetch(`http://localhost:${PORT}${path}`);
}
console.log('✅ Warm-up complete — ready for production traffic');
}
```
### Cold Start Strategy Summary
```
┌─────────────────────────────────────────────────────────────────┐
│ COLD START MITIGATION LAYERS │
│ │
│ Layer 1: Pre-Warmed Pool → 0s cold start (instant) │
│ Layer 2: Alpine Docker Image → 2s image pull (vs 12s) │
│ Layer 3: Readiness Probes → No traffic until fully ready │
│ Layer 4: Proactive Scaling → Scale before the spike │
│ Layer 5: Warm-Up Requests → Pre-establish connections │
│ │
│ Combined Effect: New containers ready in ~3-5s │
│ Without strategy: New containers ready in ~15-20s │
└─────────────────────────────────────────────────────────────────┘
```
---
## 4. Ghayma Block Storage for Stateful Workloads
### What is Ghayma Block Storage?
**Ghayma Block Storage** is a persistent, network-attached block storage service provided by the Ghayma Cloud platform. It provides raw storage volumes that can be attached to containers, similar to a virtual hard drive.
```
┌─────────────────────────────────────────────────────────────────┐
│ BLOCK STORAGE vs CONTAINER STORAGE │
│ │
│ Container (Ephemeral) Block Storage (Persistent) │
│ ┌──────────────────┐ ┌──────────────────────┐ │
│ │ Container FS │ │ Ghayma Block Volume │ │
│ │ │ │ │ │
│ │ ❌ Data lost │ │ ✅ Data survives │ │
│ │ when container│ │ container restarts│ │
│ │ stops/crashes │ │ and redeployments │ │
│ │ │ │ │ │
│ │ ❌ Not shared │ │ ✅ Can be reattached │ │
│ │ between │ │ to new containers │ │
│ │ containers │ │ │ │
│ └──────────────────┘ └──────────────────────┘ │
│ │
└─────────────────────────────────────────────────────────────────┘
```
### Why is Block Storage Needed?
Containers are **stateless and ephemeral** by design — when a container restarts, crashes, or is replaced during a deployment, all data written inside it is **lost**. This is a problem for:
| Workload Type | Data at Risk | Block Storage Solution |
| :--- | :--- | :--- |
| **Databases** (PostgreSQL, MySQL) | Table data, indexes, WAL logs | Persistent volume for `/var/lib/postgresql/data` |
| **Cache** (Redis with persistence) | RDB/AOF snapshots | Volume for `/data` |
| **File Uploads** | User-uploaded images, documents | Volume for `/app/uploads` |
| **Application Logs** | Log files for audit/compliance | Volume for `/var/log/app` |
| **Session Storage** | User session files | Volume for `/tmp/sessions` |
### Architecture: Block Storage with Containers
```
┌─────────────────────────────────────────────────────────────────────┐
│ STATEFUL WORKLOAD ARCHITECTURE │
│ │
│ ┌───────────────────┐ ┌───────────────────────────────┐ │
│ │ API Container │ │ Database Container │ │
│ │ (Stateless) │ │ (Stateful — PostgreSQL) │ │
│ │ │──────▶│ │ │
│ │ No volume │ │ ┌─────────────────────┐ │ │
│ │ needed │ │ │ /var/lib/postgresql │ │ │
│ │ │ │ │ (mount point) │ │ │
│ └───────────────────┘ │ └──────────┬──────────┘ │ │
│ │ │ │ │
│ └──────────────┼───────────────┘ │
│ │ │
│ ┌─────────▼──────────┐ │
│ │ GHAYMA BLOCK │ │
│ │ STORAGE VOLUME │ │
│ │ │ │
│ │ Name: db-data │ │
│ │ Size: 100 GB │ │
│ │ Type: SSD (gp3) │ │
│ │ IOPS: 3,000 │ │
│ │ Encrypted: Yes │ │
│ │ │ │
│ │ ✅ Persists across │ │
│ │ restarts │ │
│ │ ✅ Automatic │ │
│ │ snapshots │ │
│ │ ✅ Reattachable │ │
│ └────────────────────┘ │
│ │
└────────────────────────────────────────────────────────────────────┘
```
### How to Use Ghayma Block Storage
#### Step 1: Create a Block Storage Volume
```bash
# Using Ghayma CLI
ghaymah storage create db-data \
--size 100GB \
--type ssd \
--region me-central-1 \
--encrypted
```
#### Step 2: Attach to a Container / Deployment
```yaml
# Kubernetes PersistentVolumeClaim using Ghayma storage class
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: db-data-pvc
spec:
accessModes:
- ReadWriteOnce # Single container read/write
storageClassName: ghaymah-block-ssd
resources:
requests:
storage: 100Gi
---
# Deployment using the volume
apiVersion: apps/v1
kind: Deployment
metadata:
name: postgres-db
spec:
replicas: 1 # Stateful — typically 1 primary
template:
spec:
containers:
- name: postgres
image: postgres:16-alpine
ports:
- containerPort: 5432
volumeMounts:
- name: db-storage
mountPath: /var/lib/postgresql/data
env:
- name: POSTGRES_PASSWORD
valueFrom:
secretKeyRef:
name: db-secret
key: password
volumes:
- name: db-storage
persistentVolumeClaim:
claimName: db-data-pvc # Ghayma Block Storage volume
```
#### Step 3: Verify the Volume
```bash
# Check volume status
ghaymah storage list
# Output:
# NAME SIZE TYPE STATUS ATTACHED TO REGION
# db-data 100GB SSD attached postgres-db-pod-0 me-central-1
```
### Block Storage Features on Ghayma Cloud
| Feature | Description |
| :--- | :--- |
| **Persistence** | Data survives container restarts, crashes, and redeployments |
| **SSD Performance** | Low-latency SSD storage with up to 16,000 IOPS |
| **Encryption** | At-rest encryption (AES-256) for security compliance |
| **Snapshots** | Point-in-time snapshots for backup and disaster recovery |
| **Resizing** | Expand volume size without downtime |
| **Reattachment** | Detach from one container and attach to another (for migration) |
| **Availability Zones** | Volume and container must be in the same zone |
### When to Use Block Storage vs Other Options
| Storage Type | Use Case | Persistence | Shared Access | Speed |
| :--- | :--- | :--- | :--- | :--- |
| **Container FS** | Temp files, build artifacts | ❌ Ephemeral | ❌ No | ⚡ Fastest |
| **Ghayma Block Storage** | Databases, stateful apps, logs | ✅ Persistent | ❌ Single container | ⚡ Fast (SSD) |
| **Object Storage (S3-like)** | Images, videos, backups | ✅ Persistent | ✅ Multi-container | 🐌 Moderate |
| **Shared File Storage (NFS)** | Shared config, media libraries | ✅ Persistent | ✅ Multi-container | 🔄 Moderate |
### Data Lifecycle with Block Storage
```
Container crashes or redeploys
┌───────────────────────────────────────────┐
│ Old container is terminated │
│ ❌ Container filesystem is destroyed │
│ ✅ Block Storage volume is PRESERVED │
└────────────────────┬──────────────────────┘
┌───────────────────────────────────────────┐
│ New container starts │
│ Block Storage volume is reattached │
│ Data is immediately available at the │
│ same mount path (/var/lib/postgresql) │
│ │
│ ✅ Zero data loss │
│ ✅ Database resumes from exact state │
└───────────────────────────────────────────┘
```
---
## Summary
| Requirement | Solution |
| :--- | :--- |
| **Architecture for 15K req/s** | 3-zone architecture with L7 load balancer, 44 containers, auto-scaler, and data layer |
| **Container count** | **44 containers** (15,000 ÷ 350 effective req/s + 1 buffer) at 68.2% utilization |
| **Cold start strategy** | 5-layer approach: pre-warmed pool, Alpine images, readiness probes, proactive scaling, warm-up requests |
| **Block Storage** | Persistent SSD volumes for databases and stateful workloads that survive container restarts |
---
> **Document prepared for:** Ghayma Assessment — Scalability & Load Distribution section
> **Last updated:** July 26, 2026