feat: complete initial project structure for API and Monitoring Dashboards

scaling/auto_scaling_policy.md

@@ -0,0 +1,25 @@
+# Auto-Scaling Policy for ghaymah.systems
+
+To prevent repeating the OOMKilled outage, the platform's auto-scaling group (ASG) must be configured to respond to memory pressure as well as CPU load.
+
+## 1. Scale-Out Policy (Adding Instances)
+- **Metric:** Average Container Memory Utilization
+- **Threshold:** > 70%
+- **Evaluation Period:** 2 minutes (2 consecutive data points of 1 minute each)
+- **Action:** Add 1 container instance (Step scaling) or scale by 20% of current capacity.
+- **Cooldown Period:** 3 minutes (allows the new container to boot and start serving traffic before evaluating again).
+
+## 2. Scale-In Policy (Removing Instances)
+- **Metric:** Average Container Memory Utilization
+- **Threshold:** < 40%
+- **Evaluation Period:** 5 minutes
+- **Action:** Remove 1 container instance.
+- **Cooldown Period:** 5 minutes (prevents aggressive scale-in which might cause immediate resource pressure).
+
+## 3. CPU Backup Policy
+*Maintain existing CPU policies as a secondary trigger:*
+- Scale out if Average CPU > 75% for 2 minutes.
+
+## 4. Minimum / Maximum Capacity
+- **Min Containers:** 2 (for high availability across zones)
+- **Max Containers:** 20 (to control billing, can be adjusted based on anticipated load)

scaling/early_detection.md

@@ -0,0 +1,25 @@
+# Early Detection of Memory Issues
+
+Waiting for an application to crash (OOMKilled) is a reactive approach. To proactively detect memory issues, we must configure our monitoring tools (Prometheus, Datadog, or ghaymah metrics).
+
+## 1. High-Watermark Alerting
+Configure alerts on the metric `container_memory_usage_bytes` (or equivalent).
+
+- **Warning Alert (Slack/Teams):**
+  - Trigger: Container Memory > 80% of limit
+  - Duration: Sustained for > 3 minutes.
+  - Action: Alerts the engineering team during business hours to investigate potential memory leaks.
+
+- **Critical Alert (PagerDuty/Phone Call):**
+  - Trigger: Container Memory > 90% of limit
+  - Duration: Sustained for > 2 minutes.
+  - Action: Wakes up the on-call engineer to apply mitigations (e.g., manual scaling, restarting pods) before the crash happens.
+
+## 2. Rate of Change Alerting (Anomaly Detection)
+Sometimes memory doesn't hit a static threshold, but it grows unusually fast.
+- Monitor the *derivative* (rate of change) of memory usage.
+- If memory grows by more than 20% within 5 minutes (without a corresponding 20% spike in traffic), trigger an anomaly alert.
+
+## 3. APM Profiling
+- Integrate APM (Application Performance Monitoring) to track Garbage Collection (GC) pauses in languages like Java/Node.js, or memory footprint per request in Python/Go.
+- A sudden increase in GC time is often a precursor to an OOM event.