Compare commits
10
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
b43f9bf706 | ||
|
|
8f27b9be75 | ||
|
|
dd6d4d8be2 | ||
|
|
ce2b42226e | ||
|
|
05b671034a | ||
|
|
a072e13715 | ||
|
|
00681492f6 | ||
|
|
efd8891d0c | ||
|
|
b95a9cacfe | ||
|
|
4354ee1a32 |
@@ -0,0 +1,18 @@
|
||||
{
|
||||
"id": "3dc9f5f9-2918-4a24-b329-6414989d141d",
|
||||
"name": "exam-api-production",
|
||||
"projectId": "6d1bb957-8a2c-4a88-86f3-6a937b7971f0",
|
||||
"ports": [
|
||||
{
|
||||
"expose": true,
|
||||
"number": 5000
|
||||
}
|
||||
],
|
||||
"publicAccess": {
|
||||
"enabled": true,
|
||||
"domain": "auto"
|
||||
},
|
||||
"resourceTier": "t2",
|
||||
"dockerFileName": "Dockerfile"
|
||||
}
|
||||
|
||||
@@ -0,0 +1,18 @@
|
||||
{
|
||||
"id": "e577a6f6-fe08-41aa-9d41-0df0833e35e4",
|
||||
"name": "exam-api-staging",
|
||||
"projectId": "6d1bb957-8a2c-4a88-86f3-6a937b7971f0",
|
||||
"ports": [
|
||||
{
|
||||
"expose": true,
|
||||
"number": 5000
|
||||
}
|
||||
],
|
||||
"publicAccess": {
|
||||
"enabled": true,
|
||||
"domain": "auto"
|
||||
},
|
||||
"resourceTier": "t1",
|
||||
"dockerFileName": "Dockerfile"
|
||||
}
|
||||
|
||||
@@ -0,0 +1,96 @@
|
||||
name: CI/CD - Build & Deploy to Ghaymah Cloud
|
||||
|
||||
on:
|
||||
push:
|
||||
branches:
|
||||
- develop
|
||||
- main
|
||||
|
||||
jobs:
|
||||
# ==========================================================
|
||||
# Build & Smoke Test
|
||||
# ==========================================================
|
||||
build-and-test:
|
||||
runs-on: ubuntu-latest
|
||||
|
||||
steps:
|
||||
- name: Checkout code
|
||||
uses: actions/checkout@v5
|
||||
|
||||
- name: Build Docker image
|
||||
run: |
|
||||
docker build -t exam-api:${{ github.sha }} .
|
||||
|
||||
- name: Smoke Test
|
||||
run: |
|
||||
docker run -d -p 5000:5000 --name smoke-test exam-api:${{ github.sha }}
|
||||
sleep 8
|
||||
curl -f http://localhost:5000/health
|
||||
docker stop smoke-test
|
||||
docker rm smoke-test
|
||||
|
||||
# ==========================================================
|
||||
# Deploy Staging
|
||||
# ==========================================================
|
||||
deploy-staging:
|
||||
if: github.ref == 'refs/heads/develop'
|
||||
|
||||
needs: build-and-test
|
||||
|
||||
runs-on: ubuntu-latest
|
||||
|
||||
environment: staging
|
||||
|
||||
steps:
|
||||
- uses: actions/checkout@v5
|
||||
|
||||
- name: Install Ghaymah CLI
|
||||
run: |
|
||||
curl -sSL https://cli.ghaymah.systems/install.sh | bash
|
||||
|
||||
- name: Login
|
||||
run: |
|
||||
$HOME/ghaymah/bin/gy auth login \
|
||||
--email "${{ secrets.GHAYMAH_EMAIL }}" \
|
||||
--password "${{ secrets.GHAYMAH_PW }}"
|
||||
|
||||
- name: Use Staging Config
|
||||
run: |
|
||||
cp .ghaymah.staging.json .ghaymah.json
|
||||
|
||||
- name: Deploy to Staging
|
||||
run: |
|
||||
$HOME/ghaymah/bin/gy resource app launch
|
||||
|
||||
# ==========================================================
|
||||
# Deploy Production
|
||||
# ==========================================================
|
||||
deploy-production:
|
||||
if: github.ref == 'refs/heads/main'
|
||||
|
||||
needs: build-and-test
|
||||
|
||||
runs-on: ubuntu-latest
|
||||
|
||||
environment: production
|
||||
|
||||
steps:
|
||||
- uses: actions/checkout@v5
|
||||
|
||||
- name: Install Ghaymah CLI
|
||||
run: |
|
||||
curl -sSL https://cli.ghaymah.systems/install.sh | bash
|
||||
|
||||
- name: Login
|
||||
run: |
|
||||
$HOME/ghaymah/bin/gy auth login \
|
||||
--email "${{ secrets.GHAYMAH_EMAIL }}" \
|
||||
--password "${{ secrets.GHAYMAH_PW }}"
|
||||
|
||||
- name: Use Production Config
|
||||
run: |
|
||||
cp .ghaymah.production.json .ghaymah.json
|
||||
|
||||
- name: Deploy to Production
|
||||
run: |
|
||||
$HOME/ghaymah/bin/gy resource app launch
|
||||
@@ -0,0 +1,76 @@
|
||||
# Q4 - Scalability & Load Balancing
|
||||
|
||||
## Objective
|
||||
|
||||
Design a scalable architecture capable of handling **15,000 requests per second** on the Ghaymah Container Platform.
|
||||
|
||||
---
|
||||
|
||||
## Proposed Architecture
|
||||
|
||||
The solution uses:
|
||||
|
||||
- Ghaymah Load Balancer
|
||||
- 39 Application Containers
|
||||
- PostgreSQL Database
|
||||
- Redis Cache
|
||||
- Ghaymah Block Storage
|
||||
- Monitoring and Health Checks
|
||||
|
||||
The Load Balancer distributes incoming traffic evenly across all application containers while health checks ensure traffic is only routed to healthy instances.
|
||||
|
||||
---
|
||||
|
||||
## Container Calculation
|
||||
|
||||
Expected Traffic
|
||||
|
||||
- 15,000 requests/second
|
||||
|
||||
Container Capacity
|
||||
|
||||
- 500 requests/second
|
||||
|
||||
Required Containers
|
||||
|
||||
15000 / 500 = 30
|
||||
|
||||
Including 30% reserve capacity
|
||||
|
||||
30 × 1.3 = 39 containers
|
||||
|
||||
---
|
||||
|
||||
## Cold Start Strategy
|
||||
|
||||
The platform keeps additional warm containers available to reduce startup latency during sudden traffic spikes.
|
||||
|
||||
New containers are automatically created when resource utilization reaches the defined thresholds and are added to the Load Balancer only after passing health checks.
|
||||
|
||||
---
|
||||
|
||||
## Ghaymah Block Storage
|
||||
|
||||
Persistent storage is attached to the database layer to ensure application data survives container restarts and deployments.
|
||||
|
||||
Suitable workloads include:
|
||||
|
||||
- Databases
|
||||
- Logs
|
||||
- Uploaded files
|
||||
- Stateful applications
|
||||
|
||||
---
|
||||
|
||||
## Monitoring
|
||||
|
||||
The deployment should continuously monitor:
|
||||
|
||||
- CPU Utilization
|
||||
- Memory Usage
|
||||
- Response Time
|
||||
- Error Rate
|
||||
- Container Health
|
||||
- Restart Count
|
||||
|
||||
Alerts should be triggered before resources reach critical levels to prevent service interruption.
|
||||
@@ -0,0 +1,29 @@
|
||||
# Container Capacity Calculation
|
||||
|
||||
## Given
|
||||
|
||||
- Expected Traffic = 15,000 requests/second
|
||||
- One container capacity = 500 requests/second
|
||||
- Required headroom = 30%
|
||||
|
||||
---
|
||||
|
||||
## Step 1
|
||||
|
||||
Number of containers without headroom
|
||||
|
||||
15000 / 500 = 30 Containers
|
||||
|
||||
---
|
||||
|
||||
## Step 2
|
||||
|
||||
Adding 30% reserve capacity
|
||||
|
||||
30 × 1.30 = 39 Containers
|
||||
|
||||
---
|
||||
|
||||
## Final Result
|
||||
|
||||
The application should run **39 containers** to safely handle the expected traffic while maintaining additional capacity for sudden traffic spikes and infrastructure failures.
|
||||
Binary file not shown.
|
After Width: | Height: | Size: 1.2 MiB |
+1
-1
@@ -208,4 +208,4 @@ function formatUptime(seconds) {
|
||||
|
||||
</body>
|
||||
</html>
|
||||
|
||||
<!-- staging test -->
|
||||
|
||||
@@ -0,0 +1,72 @@
|
||||
# Q2 – Incident Postmortem
|
||||
|
||||
## Scenario
|
||||
|
||||
The production API experienced repeated **OOMKilled** events, resulting in approximately **45 minutes** of service downtime.
|
||||
|
||||
This document provides a complete incident analysis, including the timeline, root cause, resolution steps, preventive actions, an auto-scaling strategy for Ghaymah Cloud, and an early detection approach using monitoring tools.
|
||||
|
||||
---
|
||||
|
||||
## Contents
|
||||
|
||||
- Incident Summary
|
||||
- Impact Analysis
|
||||
- Timeline
|
||||
- Detection
|
||||
- Root Cause Analysis
|
||||
- Resolution
|
||||
- Lessons Learned
|
||||
- Preventive Actions
|
||||
- Ghaymah Auto-Scaling Policy
|
||||
- Early Detection Strategy
|
||||
|
||||
---
|
||||
|
||||
## Auto-Scaling on Ghaymah
|
||||
|
||||
The proposed scaling policy is designed for applications deployed on the **Ghaymah Container Platform**.
|
||||
|
||||
The application maintains at least two running replicas and automatically scales based on CPU and memory utilization.
|
||||
|
||||
Scaling thresholds:
|
||||
|
||||
- Scale Out:
|
||||
- CPU > 70%
|
||||
- Memory > 75%
|
||||
- Duration: 2 minutes
|
||||
|
||||
- Scale In:
|
||||
- CPU < 30%
|
||||
- Memory < 40%
|
||||
- Duration: 10 minutes
|
||||
|
||||
This configuration minimizes the probability of future OOMKilled incidents by distributing traffic across multiple container replicas.
|
||||
|
||||
---
|
||||
|
||||
## Monitoring Strategy
|
||||
|
||||
The application should be monitored using **Ghaymah Monitoring** together with the custom monitoring dashboard developed in Question 1.
|
||||
|
||||
The monitoring system continuously checks:
|
||||
|
||||
- Memory Usage
|
||||
- CPU Usage
|
||||
- Container Restart Count
|
||||
- OOMKilled Events
|
||||
- HTTP Response Time
|
||||
- HTTP 5xx Error Rate
|
||||
- `/health` Endpoint Status
|
||||
|
||||
Alerts are generated whenever abnormal resource utilization or unhealthy application status is detected.
|
||||
|
||||
---
|
||||
|
||||
## Conclusion
|
||||
|
||||
The incident was caused by excessive memory consumption that exceeded the container memory limit, resulting in repeated OOMKilled events.
|
||||
|
||||
The issue was resolved by rolling back to the previous stable deployment and verifying service recovery using the application's `/health` endpoint.
|
||||
|
||||
Future incidents can be mitigated through proactive monitoring, automated alerting, memory-based auto-scaling, and proper pre-production testing.
|
||||
Reference in New Issue
Block a user