Compare commits
10
Commits
7827c9aee2
...
b43f9bf706
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
b43f9bf706 | ||
|
|
8f27b9be75 | ||
|
|
dd6d4d8be2 | ||
|
|
ce2b42226e | ||
|
|
05b671034a | ||
|
|
a072e13715 | ||
|
|
00681492f6 | ||
|
|
efd8891d0c | ||
|
|
b95a9cacfe | ||
|
|
4354ee1a32 |
@@ -0,0 +1,18 @@
|
|||||||
|
{
|
||||||
|
"id": "3dc9f5f9-2918-4a24-b329-6414989d141d",
|
||||||
|
"name": "exam-api-production",
|
||||||
|
"projectId": "6d1bb957-8a2c-4a88-86f3-6a937b7971f0",
|
||||||
|
"ports": [
|
||||||
|
{
|
||||||
|
"expose": true,
|
||||||
|
"number": 5000
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"publicAccess": {
|
||||||
|
"enabled": true,
|
||||||
|
"domain": "auto"
|
||||||
|
},
|
||||||
|
"resourceTier": "t2",
|
||||||
|
"dockerFileName": "Dockerfile"
|
||||||
|
}
|
||||||
|
|
||||||
@@ -0,0 +1,18 @@
|
|||||||
|
{
|
||||||
|
"id": "e577a6f6-fe08-41aa-9d41-0df0833e35e4",
|
||||||
|
"name": "exam-api-staging",
|
||||||
|
"projectId": "6d1bb957-8a2c-4a88-86f3-6a937b7971f0",
|
||||||
|
"ports": [
|
||||||
|
{
|
||||||
|
"expose": true,
|
||||||
|
"number": 5000
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"publicAccess": {
|
||||||
|
"enabled": true,
|
||||||
|
"domain": "auto"
|
||||||
|
},
|
||||||
|
"resourceTier": "t1",
|
||||||
|
"dockerFileName": "Dockerfile"
|
||||||
|
}
|
||||||
|
|
||||||
@@ -0,0 +1,96 @@
|
|||||||
|
name: CI/CD - Build & Deploy to Ghaymah Cloud
|
||||||
|
|
||||||
|
on:
|
||||||
|
push:
|
||||||
|
branches:
|
||||||
|
- develop
|
||||||
|
- main
|
||||||
|
|
||||||
|
jobs:
|
||||||
|
# ==========================================================
|
||||||
|
# Build & Smoke Test
|
||||||
|
# ==========================================================
|
||||||
|
build-and-test:
|
||||||
|
runs-on: ubuntu-latest
|
||||||
|
|
||||||
|
steps:
|
||||||
|
- name: Checkout code
|
||||||
|
uses: actions/checkout@v5
|
||||||
|
|
||||||
|
- name: Build Docker image
|
||||||
|
run: |
|
||||||
|
docker build -t exam-api:${{ github.sha }} .
|
||||||
|
|
||||||
|
- name: Smoke Test
|
||||||
|
run: |
|
||||||
|
docker run -d -p 5000:5000 --name smoke-test exam-api:${{ github.sha }}
|
||||||
|
sleep 8
|
||||||
|
curl -f http://localhost:5000/health
|
||||||
|
docker stop smoke-test
|
||||||
|
docker rm smoke-test
|
||||||
|
|
||||||
|
# ==========================================================
|
||||||
|
# Deploy Staging
|
||||||
|
# ==========================================================
|
||||||
|
deploy-staging:
|
||||||
|
if: github.ref == 'refs/heads/develop'
|
||||||
|
|
||||||
|
needs: build-and-test
|
||||||
|
|
||||||
|
runs-on: ubuntu-latest
|
||||||
|
|
||||||
|
environment: staging
|
||||||
|
|
||||||
|
steps:
|
||||||
|
- uses: actions/checkout@v5
|
||||||
|
|
||||||
|
- name: Install Ghaymah CLI
|
||||||
|
run: |
|
||||||
|
curl -sSL https://cli.ghaymah.systems/install.sh | bash
|
||||||
|
|
||||||
|
- name: Login
|
||||||
|
run: |
|
||||||
|
$HOME/ghaymah/bin/gy auth login \
|
||||||
|
--email "${{ secrets.GHAYMAH_EMAIL }}" \
|
||||||
|
--password "${{ secrets.GHAYMAH_PW }}"
|
||||||
|
|
||||||
|
- name: Use Staging Config
|
||||||
|
run: |
|
||||||
|
cp .ghaymah.staging.json .ghaymah.json
|
||||||
|
|
||||||
|
- name: Deploy to Staging
|
||||||
|
run: |
|
||||||
|
$HOME/ghaymah/bin/gy resource app launch
|
||||||
|
|
||||||
|
# ==========================================================
|
||||||
|
# Deploy Production
|
||||||
|
# ==========================================================
|
||||||
|
deploy-production:
|
||||||
|
if: github.ref == 'refs/heads/main'
|
||||||
|
|
||||||
|
needs: build-and-test
|
||||||
|
|
||||||
|
runs-on: ubuntu-latest
|
||||||
|
|
||||||
|
environment: production
|
||||||
|
|
||||||
|
steps:
|
||||||
|
- uses: actions/checkout@v5
|
||||||
|
|
||||||
|
- name: Install Ghaymah CLI
|
||||||
|
run: |
|
||||||
|
curl -sSL https://cli.ghaymah.systems/install.sh | bash
|
||||||
|
|
||||||
|
- name: Login
|
||||||
|
run: |
|
||||||
|
$HOME/ghaymah/bin/gy auth login \
|
||||||
|
--email "${{ secrets.GHAYMAH_EMAIL }}" \
|
||||||
|
--password "${{ secrets.GHAYMAH_PW }}"
|
||||||
|
|
||||||
|
- name: Use Production Config
|
||||||
|
run: |
|
||||||
|
cp .ghaymah.production.json .ghaymah.json
|
||||||
|
|
||||||
|
- name: Deploy to Production
|
||||||
|
run: |
|
||||||
|
$HOME/ghaymah/bin/gy resource app launch
|
||||||
@@ -0,0 +1,76 @@
|
|||||||
|
# Q4 - Scalability & Load Balancing
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
|
||||||
|
Design a scalable architecture capable of handling **15,000 requests per second** on the Ghaymah Container Platform.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Proposed Architecture
|
||||||
|
|
||||||
|
The solution uses:
|
||||||
|
|
||||||
|
- Ghaymah Load Balancer
|
||||||
|
- 39 Application Containers
|
||||||
|
- PostgreSQL Database
|
||||||
|
- Redis Cache
|
||||||
|
- Ghaymah Block Storage
|
||||||
|
- Monitoring and Health Checks
|
||||||
|
|
||||||
|
The Load Balancer distributes incoming traffic evenly across all application containers while health checks ensure traffic is only routed to healthy instances.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Container Calculation
|
||||||
|
|
||||||
|
Expected Traffic
|
||||||
|
|
||||||
|
- 15,000 requests/second
|
||||||
|
|
||||||
|
Container Capacity
|
||||||
|
|
||||||
|
- 500 requests/second
|
||||||
|
|
||||||
|
Required Containers
|
||||||
|
|
||||||
|
15000 / 500 = 30
|
||||||
|
|
||||||
|
Including 30% reserve capacity
|
||||||
|
|
||||||
|
30 × 1.3 = 39 containers
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Cold Start Strategy
|
||||||
|
|
||||||
|
The platform keeps additional warm containers available to reduce startup latency during sudden traffic spikes.
|
||||||
|
|
||||||
|
New containers are automatically created when resource utilization reaches the defined thresholds and are added to the Load Balancer only after passing health checks.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Ghaymah Block Storage
|
||||||
|
|
||||||
|
Persistent storage is attached to the database layer to ensure application data survives container restarts and deployments.
|
||||||
|
|
||||||
|
Suitable workloads include:
|
||||||
|
|
||||||
|
- Databases
|
||||||
|
- Logs
|
||||||
|
- Uploaded files
|
||||||
|
- Stateful applications
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Monitoring
|
||||||
|
|
||||||
|
The deployment should continuously monitor:
|
||||||
|
|
||||||
|
- CPU Utilization
|
||||||
|
- Memory Usage
|
||||||
|
- Response Time
|
||||||
|
- Error Rate
|
||||||
|
- Container Health
|
||||||
|
- Restart Count
|
||||||
|
|
||||||
|
Alerts should be triggered before resources reach critical levels to prevent service interruption.
|
||||||
@@ -0,0 +1,29 @@
|
|||||||
|
# Container Capacity Calculation
|
||||||
|
|
||||||
|
## Given
|
||||||
|
|
||||||
|
- Expected Traffic = 15,000 requests/second
|
||||||
|
- One container capacity = 500 requests/second
|
||||||
|
- Required headroom = 30%
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Step 1
|
||||||
|
|
||||||
|
Number of containers without headroom
|
||||||
|
|
||||||
|
15000 / 500 = 30 Containers
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Step 2
|
||||||
|
|
||||||
|
Adding 30% reserve capacity
|
||||||
|
|
||||||
|
30 × 1.30 = 39 Containers
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Final Result
|
||||||
|
|
||||||
|
The application should run **39 containers** to safely handle the expected traffic while maintaining additional capacity for sudden traffic spikes and infrastructure failures.
|
||||||
Binary file not shown.
|
After Width: | Height: | Size: 1.2 MiB |
+1
-1
@@ -208,4 +208,4 @@ function formatUptime(seconds) {
|
|||||||
|
|
||||||
</body>
|
</body>
|
||||||
</html>
|
</html>
|
||||||
|
<!-- staging test -->
|
||||||
|
|||||||
@@ -0,0 +1,72 @@
|
|||||||
|
# Q2 – Incident Postmortem
|
||||||
|
|
||||||
|
## Scenario
|
||||||
|
|
||||||
|
The production API experienced repeated **OOMKilled** events, resulting in approximately **45 minutes** of service downtime.
|
||||||
|
|
||||||
|
This document provides a complete incident analysis, including the timeline, root cause, resolution steps, preventive actions, an auto-scaling strategy for Ghaymah Cloud, and an early detection approach using monitoring tools.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Contents
|
||||||
|
|
||||||
|
- Incident Summary
|
||||||
|
- Impact Analysis
|
||||||
|
- Timeline
|
||||||
|
- Detection
|
||||||
|
- Root Cause Analysis
|
||||||
|
- Resolution
|
||||||
|
- Lessons Learned
|
||||||
|
- Preventive Actions
|
||||||
|
- Ghaymah Auto-Scaling Policy
|
||||||
|
- Early Detection Strategy
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Auto-Scaling on Ghaymah
|
||||||
|
|
||||||
|
The proposed scaling policy is designed for applications deployed on the **Ghaymah Container Platform**.
|
||||||
|
|
||||||
|
The application maintains at least two running replicas and automatically scales based on CPU and memory utilization.
|
||||||
|
|
||||||
|
Scaling thresholds:
|
||||||
|
|
||||||
|
- Scale Out:
|
||||||
|
- CPU > 70%
|
||||||
|
- Memory > 75%
|
||||||
|
- Duration: 2 minutes
|
||||||
|
|
||||||
|
- Scale In:
|
||||||
|
- CPU < 30%
|
||||||
|
- Memory < 40%
|
||||||
|
- Duration: 10 minutes
|
||||||
|
|
||||||
|
This configuration minimizes the probability of future OOMKilled incidents by distributing traffic across multiple container replicas.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Monitoring Strategy
|
||||||
|
|
||||||
|
The application should be monitored using **Ghaymah Monitoring** together with the custom monitoring dashboard developed in Question 1.
|
||||||
|
|
||||||
|
The monitoring system continuously checks:
|
||||||
|
|
||||||
|
- Memory Usage
|
||||||
|
- CPU Usage
|
||||||
|
- Container Restart Count
|
||||||
|
- OOMKilled Events
|
||||||
|
- HTTP Response Time
|
||||||
|
- HTTP 5xx Error Rate
|
||||||
|
- `/health` Endpoint Status
|
||||||
|
|
||||||
|
Alerts are generated whenever abnormal resource utilization or unhealthy application status is detected.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Conclusion
|
||||||
|
|
||||||
|
The incident was caused by excessive memory consumption that exceeded the container memory limit, resulting in repeated OOMKilled events.
|
||||||
|
|
||||||
|
The issue was resolved by rolling back to the previous stable deployment and verifying service recovery using the application's `/health` endpoint.
|
||||||
|
|
||||||
|
Future incidents can be mitigated through proactive monitoring, automated alerting, memory-based auto-scaling, and proper pre-production testing.
|
||||||
Reference in New Issue
Block a user