# Scalability & Capacity Planning (15,000 req/s) — Ghaymah Cloud

**Candidate Profile**: Marwan Abdelmoneim (`marwanabdelmoneim`) | `marwantamermo@gmail.com`  
**Track**: SRE — Site Reliability Engineering  
**Platform**: Ghaymah Cloud (`ghaymah.systems`)  

---

## 1. Mathematical Capacity Calculation

### Input Parameters:
- **Target Sustained Load ($R_{\text{target}}$)**: $15,000 \text{ req/s}$
- **Single Container Throughput ($C_{\text{pod}}$)**: $500 \text{ req/s}$
- **Safety Margin Overhead ($M_{\text{safety}}$)**: $+30\%$

### Step 1: Calculate Peak Target Throughput
$$\text{Peak Capacity Requirement } (R_{\text{peak}}) = R_{\text{target}} \times (1 + M_{\text{safety}})$$
$$R_{\text{peak}} = 15,000 \times 1.30 = 19,500 \text{ req/s}$$

### Step 2: Calculate Total Required Active Containers
$$N_{\text{containers}} = \frac{R_{\text{peak}}}{C_{\text{pod}}} = \frac{19,500}{500} = \mathbf{39 \text{ active containers}}$$

### Step 3: Multi-Availability Zone Provisioning
To achieve **$N-1$ AZ Fault Tolerance** across Ghaymah Cloud's 3 Availability Zones:
- **Total Containers**: 39 Containers active
- **Per-AZ Allocation**: $\frac{39}{3} = \mathbf{13 \text{ containers per AZ}}$ (spread across `me-central-1a`, `me-central-1b`, `me-central-1c`)
- **Autoscaler Configuration**: `minReplicas: 39`, `maxReplicas: 65`.

---

## 2. Container Cold Start Mitigation Strategy

To eliminate cold start latencies when auto-scaling new containers under sudden traffic bursts:

1. **Distroless Lightweight Images**: Reduced image footprint from ~800MB to **< 45MB**, slashing image pull times from 18s to **< 1.2s**.
2. **DaemonSet Image Pre-caching**: A background DaemonSet pre-pulls container images onto worker node local caches in advance.
3. **Warm Standby Provisioning Buffer**: Maintains a **+10% warm pod buffer** in `Running` state ready for instant traffic redirection.
4. **Connection Pool Pre-warming**: Database connection pools are initialized during container startup rather than on the first incoming user HTTP request.
5. **Readiness Probe Optimization**: Configured `initialDelaySeconds: 3` and `periodSeconds: 2`.

---

## 3. Ghaymah Block Storage for Stateful Workloads

While backend application containers are 100% stateless, database stateful workloads (PostgreSQL / Redis) utilize Ghaymah Block Storage:

```
┌─────────────────────────────────────────────────────────────┐
│                 StatefulSet Database Pod                    │
└──────────────────────────────┬──────────────────────────────┘
                               │ PVC Mount: /var/lib/postgresql/data
┌──────────────────────────────▼──────────────────────────────┐
│  Ghaymah Block Storage Volume (ghaymah-block-nvme SSD)      │
│  • Provisioned Performance: 12,000 IOPS / 500 MB/s          │
│  • Access Mode: ReadWriteOnce (RWO)                         │
│  • Snapshots: Automated Hourly Snapshots to Object Storage  │
└─────────────────────────────────────────────────────────────┘
```

1. **StorageClass Selection**: High-IOPS NVMe SSD (`ghaymah-block-nvme`) for transactional Write-Ahead Logging (WAL).
2. **Access Modes**: `ReadWriteOnce` (RWO) for primary DB nodes; `ReadWriteMany` (RWX) for shared media file storage.
3. **Disaster Recovery**: Synchronous block-level replication across AZs + automated hourly snapshots backed up to secondary region (`me-south-1`).