Compare commits

..
10 Commits
19 changed files with 1146 additions and 2 deletions
+18
View File
@@ -0,0 +1,18 @@
name: Docker Build
on:
push:
branches:
- main
jobs:
docker:
runs-on: ubuntu-latest
steps:
- name: Checkout Repository
uses: actions/checkout@v4
- name: Build Docker Image
run: |
docker build -t ghaymah-sre-api:latest .
+2 -2
View File
@@ -1,8 +1,8 @@
from flask import Flask, jsonify from flask import Flask, jsonify
from datetime import datetime from datetime import datetime
from flask_cors import CORS
app = Flask(__name__) app = Flask(__name__)
CORS(app)
request_count = 0 request_count = 0
@@ -0,0 +1,111 @@
# Integration Proposal 1
# Speaking Genie + Ghaymah Systems
## Product Information
**Product:** Speaking Genie
**Official Website:**
https://speakingenie.com/
**Mortakaz Listing:**
https://www.mortakaz.com/updates
---
# Product Description
Speaking Genie is an educational platform that helps users improve their English speaking skills using interactive AI-powered lessons and conversations. It focuses on making language learning practical, engaging, and accessible.
---
# Proposed Integration with Ghaymah Systems
Speaking Genie can leverage Ghaymah Cloud to improve scalability, reliability, and deployment.
### Proposed Services
- Ghaymah Container Platform
- Ghaymah Container Registry
- Ghaymah Block Storage
- Auto Scaling
- Load Balancer
Application updates can be deployed automatically using GitHub Actions and Docker containers.
---
# Integration with mithal.space
Mithal.space can continuously monitor:
- HTTP Availability
- Response Time
- SSL Certificate
- DNS Resolution
- API/Search Response
Alerts can be generated whenever service quality decreases.
---
# Benefits for End Users
- Faster application performance
- Higher availability
- Automatic scaling during peak traffic
- Secure HTTPS deployment
- Continuous monitoring
- Reduced downtime
---
# Architecture Sketch
```
Students
|
|
HTTPS Requests
|
v
+-------------------+
| ghaymah.systems |
| Load Balancer |
+---------+---------+
|
+---------+---------+
| |
Container 1 Container 2
| |
+---------+---------+
|
Block Storage
|
Speaking Genie
|
mithal.space Monitoring
```
---
# Technical Challenges
- Session persistence
- Scaling AI workloads
- Database optimization
- Storage performance
---
# Business Challenges
- Infrastructure cost
- Customer migration
- Data privacy compliance
---
# Conclusion
Deploying Speaking Genie on Ghaymah would provide a highly available, scalable, and monitored environment while improving the user experience.
@@ -0,0 +1,108 @@
# Integration Proposal 2
# Crawleo + Ghaymah Systems
## Product Information
**Product:** Crawleo
**Mortakaz Listing:**
https://www.mortakaz.com/updates
---
# Product Description
Crawleo is a digital platform that helps users analyze and organize website information through automated crawling and data collection. It simplifies large-scale website analysis and data extraction.
---
# Proposed Integration with Ghaymah Systems
Crawleo can run as a containerized application on Ghaymah Cloud.
### Proposed Services
- Docker Containers
- Container Registry
- Auto Scaling
- Load Balancer
- Block Storage
Multiple crawler containers can execute jobs simultaneously while sharing persistent storage.
---
# Integration with mithal.space
Mithal.space can monitor:
- HTTP Availability
- API Response Time
- SSL Status
- DNS Resolution
- Crawl Service Health
This enables administrators to detect failures quickly.
---
# Benefits for End Users
- Faster crawling
- Better reliability
- Automatic scaling
- High availability
- Centralized monitoring
- Persistent storage for crawl results
---
# Architecture Sketch
```
Users
|
|
HTTPS Requests
|
v
+--------------------+
| ghaymah.systems |
| Load Balancer |
+---------+----------+
|
+-----------+-----------+
| |
Crawler Container 1 Crawler Container 2
| |
+-----------+-----------+
|
Block Storage
|
Crawl Database
|
mithal.space Monitoring
```
---
# Technical Challenges
- Distributed crawling coordination
- Storage optimization
- Duplicate request handling
- Horizontal scaling
---
# Business Challenges
- Infrastructure cost
- API rate limits
- Compliance with website policies
---
# Recommendation
Crawleo would greatly benefit from Ghaymah's container platform because crawling workloads are naturally parallel and can scale horizontally. Combined with mithal.space monitoring, administrators can ensure continuous availability and quickly detect service issues.
+1
View File
@@ -0,0 +1 @@
https://qabilah.com/profile/hassan90122/posts
+34
View File
@@ -0,0 +1,34 @@
<!DOCTYPE html>
<html>
<head>
<title>Ghaymah SRE Dashboard</title>
<link rel="stylesheet" href="style.css">
</head>
<body>
<h1>Ghaymah SRE Dashboard</h1>
<div class="card">
<h2>Status</h2>
<p id="status">Checking...</p>
<h2>Response Time</h2>
<p id="response">-</p>
<h2>Total Requests</h2>
<p id="requests">-</p>
</div>
<script src="script.js"></script>
</body>
</html>
+67
View File
@@ -0,0 +1,67 @@
const API_URL =
"https://ghaymah-sre-api-20e343cababc.hosted.ghaymah.systems";
async function checkApplication(){
try {
let start = Date.now();
let healthResponse =
await fetch(API_URL + "/health");
let end = Date.now();
let responseTime =
end - start;
let statsResponse =
await fetch(API_URL + "/stats");
let stats =
await statsResponse.json();
document.getElementById("status").innerHTML =
healthResponse.ok ? "UP" : "DOWN";
document.getElementById("response").innerHTML =
responseTime + " ms";
document.getElementById("requests").innerHTML =
stats.requests;
}
catch(error){
document.getElementById("status").innerHTML =
"DOWN";
console.log(error);
}
}
checkApplication();
setInterval(
checkApplication,
30000
);
+32
View File
@@ -0,0 +1,32 @@
body {
font-family: Arial, sans-serif;
text-align: center;
background: #f4f4f4;
}
h1 {
margin-top: 40px;
}
.card {
background: white;
width: 350px;
margin: 40px auto;
padding: 25px;
border-radius: 10px;
}
p {
font-size: 22px;
}
+32
View File
@@ -0,0 +1,32 @@
# Staging vs Production
## Staging Environment
Purpose:
- Testing before release.
- Validate new features.
- Detect bugs.
- Verify Docker images.
- Run integration tests.
Characteristics:
- Mirrors production.
- Uses test data.
- Safe for experimentation.
- Developers and QA have access.
---
## Production Environment
Purpose:
- Serve real users.
- Provide stable services.
- High availability.
- Performance optimized.
Characteristics:
- Uses real data.
- Restricted access.
- Monitored continuously.
- Requires approval before deployment.
+56
View File
@@ -0,0 +1,56 @@
# Ghaymah CLI Integration
## Install the CLI
```bash
curl -fsSL https://ghaymah.systems/install.sh | bash
```
---
## Login
```bash
ghaymah login
```
---
## Verify Login
```bash
ghaymah whoami
```
---
## Build Docker Image
```bash
docker build -t ghaymah-sre-api .
```
---
## Push Image
```bash
docker push registry.ghaymah.systems/<username>/ghaymah-sre-api:latest
```
---
## Deploy
```bash
ghaymah deploy \
--image registry.ghaymah.systems/<username>/ghaymah-sre-api:latest
```
---
## Check Deployment
```bash
ghaymah apps list
```
+195
View File
@@ -0,0 +1,195 @@
# Incident Postmortem: Repeated OOMKilled Causing Application Downtime
## Incident Summary
**Incident Title:** Repeated OOMKilled Causing Application Downtime
**Date:** July 28, 2026
**Duration:** 45 Minutes
**Severity:** High
**Affected Service:** Ghaymah SRE API
### Impact
- The application was unavailable for approximately 45 minutes.
- Users experienced failed requests and service interruptions.
- Multiple container restarts occurred due to repeated OOMKilled events.
---
# Timeline
| Time | Event |
|------|-------|
| 10:00 | Application deployed successfully |
| 10:08 | Memory usage started increasing rapidly |
| 10:12 | First OOMKilled event occurred |
| 10:13 - 10:40 | Container repeatedly restarted due to memory exhaustion |
| 10:20 | Monitoring system generated alerts |
| 10:25 | SRE team started investigation |
| 10:35 | Memory leak identified |
| 10:42 | Memory limit increased and deployment restarted |
| 10:45 | Service fully recovered |
---
# Root Cause Analysis
The application experienced a memory leak that continuously increased memory consumption.
Once the container exceeded its configured memory limit, Kubernetes terminated it with an **OOMKilled** event.
Because the underlying issue was not resolved immediately, the container repeatedly restarted, causing approximately 45 minutes of service downtime.
---
# Recommendations
## Immediate Actions
- Increase the container memory limit.
- Restart the deployment after applying the fix.
- Monitor memory usage during recovery.
## Long-Term Improvements
- Configure Horizontal Pod Autoscaler (HPA).
- Set appropriate CPU and memory requests and limits.
- Enable memory utilization alerts.
- Monitor container restart counts.
- Perform load testing before production deployments.
- Regularly profile the application to detect memory leaks.
---
# Auto-Scaling Policy
## Objective
Automatically scale the application based on workload to maintain availability and reduce the risk of resource exhaustion.
| Configuration | Value |
|--------------|-------|
| Minimum Replicas | 2 |
| Maximum Replicas | 10 |
| CPU Target | 70% |
| Memory Target | 75% |
| Scale Up | Add 1–2 replicas after 2 minutes above threshold |
| Scale Down | Remove 1 replica after 10 minutes of low utilization |
## Scale-Up Rules
Trigger scaling when:
- CPU usage exceeds 70%.
- Memory usage exceeds 75%.
- Traffic increases significantly.
Action:
- Create additional application replicas.
- Distribute requests using the load balancer.
## Scale-Down Rules
Trigger scaling when:
- CPU remains below 30%.
- Memory remains below 40%.
- Low traffic continues for at least 10 minutes.
Action:
- Gradually remove unused replicas while keeping at least two running.
---
# Early Detection Using Monitoring
## Monitoring Metrics
### Memory Usage
Monitor:
- Container memory usage
- Memory utilization percentage
- Available memory
**Alert Rule**
```
Memory usage > 80% for 5 minutes
```
### OOMKilled Events
Monitor:
- Number of OOMKilled events
- Container restart count
**Alert Rule**
```
Restart count > 3 within 10 minutes
```
### Container Health
Monitor:
- /health endpoint
- Readiness probe
- Liveness probe
- Pod status
Alert if:
- Health endpoint returns a non-200 status.
- Pods enter CrashLoopBackOff or Pending state.
### Application Performance
Monitor:
- HTTP response time
- Request rate
- Error rate (5xx)
- Active requests
---
# Monitoring Dashboard
The dashboard should display:
- Application Status
- CPU Usage
- Memory Usage
- Response Time
- Total Requests
- Container Restarts
- OOMKilled Events
- Running Replicas
---
# Alert Workflow
1. Prometheus collects application and container metrics.
2. Alertmanager evaluates alert rules.
3. Notifications are sent to the SRE team (Email, Slack, or Microsoft Teams).
4. Engineers investigate the issue using Grafana dashboards and application logs.
5. If auto-scaling is enabled, additional replicas are created automatically while the issue is being investigated.
---
# Conclusion
The outage was caused by excessive memory consumption that resulted in repeated OOMKilled events and continuous container restarts.
Implementing proactive monitoring, alerting, resource limits, and auto-scaling policies will significantly reduce the likelihood and impact of similar incidents in the future.
+49
View File
@@ -0,0 +1,49 @@
import requests
import time
from datetime import datetime
URL = "https://ghaymah-sre-api-20e343cababc.hosted.ghaymah.systems/health"
while True:
try:
start = time.time()
response = requests.get(URL)
end = time.time()
response_time = round(end - start, 3)
if response.status_code == 200:
print(
datetime.now(),
"UP",
"Response time:",
response_time,
"seconds"
)
else:
print(
datetime.now(),
"DOWN",
response.status_code
)
except Exception as e:
print(
datetime.now(),
"ERROR",
e
)
time.sleep(30)
+179
View File
@@ -0,0 +1,179 @@
<!DOCTYPE html>
<html>
<head>
<meta charset="UTF-8">
<title>Mithal Monitoring Dashboard</title>
<script src="https://cdn.jsdelivr.net/npm/chart.js"></script>
<style>
body{
font-family:Arial;
background:#f4f4f4;
text-align:center;
}
.card{
background:white;
padding:20px;
margin:20px auto;
width:90%;
max-width:900px;
border-radius:10px;
box-shadow:0 0 10px rgba(0,0,0,.15);
}
table{
width:100%;
border-collapse:collapse;
}
th,td{
border:1px solid #ddd;
padding:10px;
}
</style>
</head>
<body>
<h1>Mithal Monitoring Dashboard</h1>
<div class="card">
<h2>Uptime (24 Hours)</h2>
<p id="uptime">Loading...</p>
</div>
<div class="card">
<h2>SSL Certificate</h2>
<p id="ssl">Loading...</p>
</div>
<div class="card">
<h2>Latency (Last Hour)</h2>
<canvas id="chart"></canvas>
</div>
<div class="card">
<h2>Last 10 Checks</h2>
<table id="table"></table>
</div>
<script>
fetch("metrics.csv")
.then(r=>r.text())
.then(text=>{
let rows=text.trim().split("\n").slice(1);
let labels=[];
let latency=[];
let success=0;
let ssl=0;
let table="<tr><th>Time</th><th>Status</th><th>Latency</th></tr>";
rows.forEach(row=>{
let c=row.split(",");
labels.push(c[0]);
latency.push(Number(c[1]));
ssl=c[3];
if(c[2]=="200") success++;
});
document.getElementById("uptime").innerHTML=
((success/rows.length)*100).toFixed(2)+" %";
document.getElementById("ssl").innerHTML=
ssl+" days remaining";
rows.slice(-10).forEach(row=>{
let c=row.split(",");
table+=`<tr>
<td>${c[0]}</td>
<td>${c[2]}</td>
<td>${c[1]} ms</td>
</tr>`;
});
document.getElementById("table").innerHTML=table;
new Chart(document.getElementById("chart"),{
type:"line",
data:{
labels:labels.slice(-60),
datasets:[{
label:"Latency (ms)",
data:latency.slice(-60)
}]
}
});
});
</script>
</body>
</html>
+2
View File
@@ -0,0 +1,2 @@
timestamp,latency_ms,status_code,ssl_days_left,dns_ms,search_ms
2026-07-28 21:48:04,1089.54,200,48,3.73,9141.39
1 timestamp latency_ms status_code ssl_days_left dns_ms search_ms
2 2026-07-28 21:48:04 1089.54 200 48 3.73 9141.39
+94
View File
@@ -0,0 +1,94 @@
import csv
import os
import time
import ssl
import socket
from datetime import datetime
import requests
import dns.resolver
from OpenSSL import crypto
URL = "https://mithal.space"
SEARCH_URL = "https://mithal.space"
CSV_FILE = "metrics.csv"
if not os.path.exists(CSV_FILE):
with open(CSV_FILE, "w", newline="") as file:
writer = csv.writer(file)
writer.writerow([
"timestamp",
"latency_ms",
"status_code",
"ssl_days_left",
"dns_ms",
"search_ms"
])
while True:
timestamp = datetime.utcnow().strftime("%Y-%m-%d %H:%M:%S")
# HTTP Latency + Uptime
try:
start = time.time()
response = requests.get(URL, timeout=10)
latency = round((time.time() - start) * 1000, 2)
status = response.status_code
except:
latency = -1
status = 0
# SSL
try:
cert = ssl.get_server_certificate(("mithal.space", 443))
cert = crypto.load_certificate(
crypto.FILETYPE_PEM,
cert
)
expire = datetime.strptime(
cert.get_notAfter().decode(),
"%Y%m%d%H%M%SZ"
)
ssl_days = (expire - datetime.utcnow()).days
except:
ssl_days = -1
# DNS
try:
start = time.time()
dns.resolver.resolve("mithal.space", "A")
dns_time = round((time.time() - start) * 1000, 2)
except:
dns_time = -1
# Search
try:
start = time.time()
requests.get(
SEARCH_URL,
params={"q": "cloud"},
timeout=10
)
search_time = round((time.time() - start) * 1000, 2)
except:
search_time = -1
with open(CSV_FILE, "a", newline="") as file:
writer = csv.writer(file)
writer.writerow([
timestamp,
latency,
status,
ssl_days,
dns_time,
search_time
])
print(timestamp, status, latency)
time.sleep(60)
Binary file not shown.

After

Width:  |  Height:  |  Size: 28 KiB

+33
View File
@@ -0,0 +1,33 @@
# Scalability Calculation
## Given
Incoming traffic = 15,000 requests/second
Container capacity = 500 requests/second
Safety margin = 30%
---
## Step 1
Minimum containers required:
15000 / 500 = 30 containers
---
## Step 2
Add a 30% safety margin:
30 × 1.30 = 39 containers
---
## Final Result
Required containers = **39**
This deployment can safely handle 15,000 requests per second while providing additional capacity for traffic spikes and failover.
+70
View File
@@ -0,0 +1,70 @@
# Cold Start Strategy for New Containers
## Overview
A cold start occurs when the platform launches a new container because application traffic has increased and additional capacity is required.
## Strategy
### 1. Keep Warm Instances
Maintain a small number of idle containers (2–3 replicas) that are always running and ready to receive traffic immediately.
---
### 2. Horizontal Auto Scaling
Automatically create additional containers when one or more of the following conditions are met:
- CPU usage exceeds 70%
- Memory usage exceeds 75%
- Request rate increases significantly
- Response time exceeds the defined threshold
---
### 3. Fast Container Startup
Reduce container startup time by:
- Using lightweight Docker images
- Installing only required packages
- Minimizing application initialization
- Preloading configuration files
---
### 4. Health Checks
A new container should not receive production traffic until it successfully passes:
- Readiness Probe
- Health Check
This ensures users are only routed to healthy containers.
---
### 5. Rolling Updates
Deploy new application versions gradually to avoid downtime.
Example:
Old Version
██████████
↓
New Version
██░░░░░░░░
↓
████░░░░░
↓
██████████
This strategy guarantees continuous service availability during deployments.
+63
View File
@@ -0,0 +1,63 @@
# Using Ghaymah Block Storage for Stateful Workloads
## Overview
Ghaymah Block Storage provides persistent storage that remains available even if containers are restarted, replaced, or rescheduled.
It is designed for applications that must preserve data.
## Common Stateful Workloads
- PostgreSQL
- MySQL
- MongoDB
- Elasticsearch
- Jenkins
- GitLab
- File Storage
- Application Uploads
---
## Benefits
### Persistent Data
Data remains available after:
- Container restart
- Node failure
- Deployment updates
---
### High Performance
Block Storage provides low-latency, high-performance storage suitable for databases and transactional workloads.
---
### Easy Scaling
Storage volumes can be expanded without recreating the application.
---
### Reliability
Storage is independent of containers, reducing the risk of data loss and improving application resilience.
---
## Example Architecture
Application Container
|
|
Persistent Volume
|
|
Ghaymah Block Storage
|
|
PostgreSQL Database