diff --git a/q1-deploy-monitor/health-check.sh b/q1-deploy-monitor/health-check.sh index fcff5ed..6eef70d 100755 --- a/q1-deploy-monitor/health-check.sh +++ b/q1-deploy-monitor/health-check.sh @@ -1,6 +1,6 @@ #!/usr/bin/env bash -LOG_FILE="/Users/mac/a1/monitor/monitor.log" +LOG_FILE="/Users/mac/a1/q1-deploy-monitor/monitor.log" # Ensure log file exists mkdir -p "$(dirname "$LOG_FILE")" diff --git a/q2-postmortem/postmortem-report.md b/q2-postmortem/postmortem-report.md index e06e4af..5d53bd8 100644 --- a/q2-postmortem/postmortem-report.md +++ b/q2-postmortem/postmortem-report.md @@ -26,3 +26,15 @@ - **Short-term:** Implement a memory-based auto-scaling rule (Scale out when Memory > 70%). - **Medium-term:** Setup early-detection alerts for memory utilization reaching 80% to warn the team *before* an OOMKilled event occurs. - **Long-term:** Profile the application to identify memory bottlenecks or leaks. + +## 5. Auto-Scaling Policy +To prevent repeating this incident, the platform's auto-scaling group must be configured as follows: +- **Scale-Out Policy:** Add 1 container instance when Average Container Memory > 70% for 2 minutes. +- **Scale-In Policy:** Remove 1 container instance when Average Container Memory < 40% for 5 minutes. +- **CPU Backup Policy:** Scale out if Average CPU > 75% for 2 minutes. + +## 6. Early Detection +To detect this issue before the container crashes: +- Configure alerts on the metric `container_memory_usage_bytes` (or equivalent). +- **Warning Alert:** Container Memory > 80% (Sustained for > 3 minutes) - triggers Slack/Teams notification. +- **Critical Alert:** Container Memory > 90% (Sustained for > 2 minutes) - triggers PagerDuty to wake up on-call engineer for immediate manual mitigation. diff --git a/q3-cicd/workflow.yml b/q3-cicd/workflow.yml index da09dc7..386cfda 100644 --- a/q3-cicd/workflow.yml +++ b/q3-cicd/workflow.yml @@ -58,3 +58,20 @@ jobs: run: | echo "Deploying production-${{ github.sha }} to ghaymah systems..." # Example CLI command: ghaymah deploy --image ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}:production-${{ github.sha }} --env production + +# ===================================================================== +# Q3 Additional Requirements Documentation +# ===================================================================== +# +# 3. Difference between Staging and Production: +# - Staging (`staging`): A pre-production environment used for QA and integration testing. +# It uses sanitized/dummy data and is scaled down. Deploys happen automatically upon merging to main. +# - Production (`production`): The live environment for real users. Contains sensitive, live data. +# Requires strict access controls and a **Manual Approval** step in GitHub Actions before deployment. +# +# 4. Integration with Ghaymah CLI: +# To deploy manually or from a runner using the CLI: +# 1. Install: `curl -sL https://cli.ghaymah.systems/install.sh | bash` +# 2. Login: `ghaymah login --token $GHAYMAH_TOKEN` +# 3. Push: `docker push registry.ghaymah.systems/my-org/myapp-api:latest` +# 4. Deploy: `ghaymah deploy --name myapp-api --image registry.ghaymah.systems/my-org/myapp-api:latest --port 8080 --env production` diff --git a/q4-scalability/calculations.md b/q4-scalability/calculations.md index 6e4c26d..a31d0a3 100644 --- a/q4-scalability/calculations.md +++ b/q4-scalability/calculations.md @@ -22,3 +22,15 @@ To ensure high availability and responsiveness under a load of 15,000 requests p ## Conclusion To safely handle 15,000 req/s while maintaining a 30% safety margin (which helps absorb sudden traffic spikes or the failure of a few containers), the auto-scaling group should be configured to maintain a baseline of **43 containers** during peak load. + +## 4. Cold Start Strategy +To minimize the delay when new containers are provisioned (cold start latency): +1. **Lightweight Base Images:** Use Alpine or distroless images (e.g., `python:3.11-alpine`) so they pull faster over the network. +2. **Pre-warming (Buffer Pool):** Maintain a buffer of idle containers (e.g., 10% of required capacity). For 43 containers, run ~47. The extra 4 handle sudden spikes instantly. +3. **Lazy Loading:** Defer non-critical initialization until after the container has started accepting requests. + +## 5. Ghaymah Block Storage for Stateful Workloads +While the API is mostly stateless, Ghaymah Block Storage is used for: +- **Local Caching / ML Models:** Persistent storage for large datasets downloaded at startup. +- **Session Data / Logs:** Persisting complex audit logs before they are shipped to centralized logging. +- **Self-Managed Databases:** Ensuring data survives container restarts by mounting a volume like `/mnt/data`. diff --git a/q5-mithal-monitor/dashboard.html b/q5-mithal-monitor/dashboard.html index a399613..f5fc93c 100644 --- a/q5-mithal-monitor/dashboard.html +++ b/q5-mithal-monitor/dashboard.html @@ -159,5 +159,22 @@ } tbody.innerHTML = rows; + + diff --git a/q5-mithal-monitor/data/metrics.json b/q5-mithal-monitor/data/metrics.json new file mode 100644 index 0000000..015be50 --- /dev/null +++ b/q5-mithal-monitor/data/metrics.json @@ -0,0 +1,42 @@ +[ + { + "timestamp": "2026-07-27T11:27:47.907297Z", + "latency_ms": 601.08, + "status_code": 200, + "uptime": true, + "ssl_status": "Error: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1081)", + "ssl_days": 0, + "dns_time_ms": 354.55, + "search_time_ms": 637.0 + }, + { + "timestamp": "2026-07-27T11:28:50.010086Z", + "latency_ms": 627.2, + "status_code": 200, + "uptime": true, + "ssl_status": "Error: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1081)", + "ssl_days": 0, + "dns_time_ms": 64.76, + "search_time_ms": 599.89 + }, + { + "timestamp": "2026-07-27T11:29:51.597790Z", + "latency_ms": 635.14, + "status_code": 200, + "uptime": true, + "ssl_status": "Error: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1081)", + "ssl_days": 0, + "dns_time_ms": 86.34, + "search_time_ms": 721.88 + }, + { + "timestamp": "2026-07-27T11:30:53.354004Z", + "latency_ms": 596.43, + "status_code": 200, + "uptime": true, + "ssl_status": "Error: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1081)", + "ssl_days": 0, + "dns_time_ms": 71.37, + "search_time_ms": 605.94 + } +] \ No newline at end of file diff --git a/q5-mithal-monitor/mithal.log b/q5-mithal-monitor/mithal.log new file mode 100644 index 0000000..10d7111 --- /dev/null +++ b/q5-mithal-monitor/mithal.log @@ -0,0 +1,2 @@ +/Users/mac/a1/q5-mithal-monitor/monitor.py:36: DeprecationWarning: datetime.datetime.utcnow() is deprecated and scheduled for removal in a future version. Use timezone-aware objects to represent datetimes in UTC: datetime.datetime.now(datetime.UTC). + "timestamp": datetime.utcnow().isoformat() + "Z", diff --git a/q5-mithal-monitor/monitor.py b/q5-mithal-monitor/monitor.py index 95f4353..2561db8 100644 --- a/q5-mithal-monitor/monitor.py +++ b/q5-mithal-monitor/monitor.py @@ -9,7 +9,7 @@ import dns.resolver TARGET_URL = "https://mithal.space" TARGET_DOMAIN = "mithal.space" -DATA_FILE = "/Users/mac/a1/mithal_monitor/data/metrics.json" +DATA_FILE = "/Users/mac/a1/q5-mithal-monitor/data/metrics.json" def check_ssl(domain): context = ssl.create_default_context()