feat: initialize repository with SRE infrastructure, monitoring dashboard, and deployment documentation
230
README.md
Normal file
@@ -0,0 +1,230 @@
|
|||||||
|
# 🚀 Ghaymah SRE Practical Exam
|
||||||
|
|
||||||
|
**Candidate Name:** Ahmed Abdelaziz Hussein
|
||||||
|
**Track:** Site Reliability Engineering (SRE)
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 📁 Repository Structure
|
||||||
|
|
||||||
|
Below is the directory tree of the submission, showing all implementation files:
|
||||||
|
|
||||||
|
```text
|
||||||
|
.
|
||||||
|
├── common-qabilah
|
||||||
|
│ └── qabilah-profile.txt
|
||||||
|
├── q1-deploy-monitor
|
||||||
|
│ ├── Dockerfile
|
||||||
|
│ ├── dashboard.html
|
||||||
|
│ ├── health-check.sh
|
||||||
|
│ └── status.json
|
||||||
|
├── q2-postmortem
|
||||||
|
│ └── postmortem-report.md
|
||||||
|
├── q3-cicd
|
||||||
|
│ └── workflow.yml
|
||||||
|
├── q4-sacalability
|
||||||
|
│ ├── architecture.png
|
||||||
|
│ └── calculations.md
|
||||||
|
└── q5-mithal-monitor
|
||||||
|
├── dashbourd.html
|
||||||
|
├── metrics.json
|
||||||
|
└── monitor.py
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 🔧 Technologies Used
|
||||||
|
|
||||||
|
The project leverages the following technologies and frameworks:
|
||||||
|
- **Go / Fiber** (API Development)
|
||||||
|
- **Docker** (Containerization & Multi-stage builds)
|
||||||
|
- **GitHub Actions** (CI/CD Pipeline)
|
||||||
|
- **HTML / CSS / JavaScript** (Dashboards & Visualization)
|
||||||
|
- **Python** (Mithal Automated Monitoring)
|
||||||
|
- **Bash** (Monitoring Agent scripts)
|
||||||
|
- **Ghaymah Cloud** (Deployment Infrastructure & CLI)
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 📊 Question 1 – Deploy Application & Monitoring
|
||||||
|
|
||||||
|
### Requirements Covered
|
||||||
|
| Requirement | Status | Details |
|
||||||
|
| :--- | :---: | :--- |
|
||||||
|
| Dockerfile | ✔ | Optimized multi-stage build starting from `golang:alpine` to `scratch` |
|
||||||
|
| Simple Go Fiber API | ✔ | REST API utilizing the Go Fiber framework |
|
||||||
|
| `/health` endpoint | ✔ | Returns application health status and current timestamp |
|
||||||
|
| `/metrics` endpoint | ✔ | Exposes real-time internal metrics (uptime, request count, latency) |
|
||||||
|
| Monitoring Script | ✔ | Bash script performing periodic checks and exporting metrics to JSON |
|
||||||
|
| Monitoring Dashboard | ✔ | Static HTML dashboard to display real-time statuses and metrics |
|
||||||
|
| Deployment on Ghaymah Cloud | ✔ | API deployed live on Ghaymah systems |
|
||||||
|
|
||||||
|
### Files
|
||||||
|
- [`q1-deploy-monitor/Dockerfile`](./q1-deploy-monitor/Dockerfile)
|
||||||
|
- [`q1-deploy-monitor/health-check.sh`](./q1-deploy-monitor/health-check.sh)
|
||||||
|
- [`q1-deploy-monitor/dashboard.html`](./q1-deploy-monitor/dashboard.html)
|
||||||
|
- [`q1-deploy-monitor/status.json`](./q1-deploy-monitor/status.json)
|
||||||
|
|
||||||
|
### Proof of Implementation
|
||||||
|
The screenshots showcasing the running state are located under `docs/screenshots/`:
|
||||||
|
|
||||||
|
1. **Deployment on Ghaymah Cloud**
|
||||||
|

|
||||||
|
|
||||||
|
2. **Monitoring Script Running**
|
||||||
|

|
||||||
|
|
||||||
|
3. **Dashboard**
|
||||||
|

|
||||||
|
---
|
||||||
|
|
||||||
|
## 📑 Question 2 – Incident Postmortem
|
||||||
|
|
||||||
|
### Requirements Covered
|
||||||
|
- [x] **Executive Summary** (Incident metadata, downtime duration, root cause, impact)
|
||||||
|
- [x] **Timeline** (Detailed chronological sequence of events from spike to resolution)
|
||||||
|
- [x] **Root Cause Analysis (RCA)** (Detailed diagnosis of resource limits and missing HPA)
|
||||||
|
- [x] **Recommendations** (Immediate P0 fixes and long-term action items)
|
||||||
|
- [x] **Auto Scaling Policy** (Configured Horizontal Pod Autoscaler YAML config for Ghanimah)
|
||||||
|
- [x] **Monitoring Strategy** (Prometheus alert rules for early warning and OOM checks)
|
||||||
|
|
||||||
|
### Files
|
||||||
|
- [`q2-postmortem/postmortem-report.md`](file:///home/ahmed/ghaymah_task/ghaymah-exam-ahmed-abdelaziz-SRE/q2-postmortem/postmortem-report.md)
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## ⚙️ Question 3 – CI/CD Pipeline
|
||||||
|
|
||||||
|
### Requirements Covered
|
||||||
|
- [x] **Docker Build:** Builds container images on triggers.
|
||||||
|
- [x] **Container Registry:** Image tagging for version control.
|
||||||
|
- [x] **Deploy to Staging:** Triggered automatically on release branches.
|
||||||
|
- [x] **Manual Approval:** Gatekeeping promotion using GitHub Environments rules.
|
||||||
|
- [x] **Deploy to Production:** Triggered on main branch post-approval.
|
||||||
|
- [x] **Ghaymah CLI Integration:** Automated login and app launch in workflow jobs.
|
||||||
|
|
||||||
|
### Deployment Flow Diagram
|
||||||
|
```text
|
||||||
|
Push
|
||||||
|
↓
|
||||||
|
Build
|
||||||
|
↓
|
||||||
|
Deploy Staging
|
||||||
|
↓
|
||||||
|
Manual Approval
|
||||||
|
↓
|
||||||
|
Deploy Production
|
||||||
|
```
|
||||||
|
|
||||||
|
### Files
|
||||||
|
- [`q3-cicd/workflow.yml`](./q3-cicd/workflow.yml)
|
||||||
|
|
||||||
|
### Proof of Implementation
|
||||||
|
1. **Staging Deployment**
|
||||||
|

|
||||||
|
|
||||||
|
2. **Manual Approval**
|
||||||
|

|
||||||
|
|
||||||
|
3. **Production Deployment**
|
||||||
|

|
||||||
|
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 📈 Question 4 – Scalability & Load Balancing
|
||||||
|
|
||||||
|
### Requirements Covered
|
||||||
|
- [x] **Architecture Diagram** (Visual flow of requests to handle 15,000 req/s across zones)
|
||||||
|
- [x] **Capacity Calculation** (Calculations justifying 39 Pods based on limits and 30% safety buffer)
|
||||||
|
- [x] **Cold Start Strategy** (Multi-stage scratch image design, pre-warming, and probe tuning)
|
||||||
|
- [x] **Block Storage** (Dynamic PV/PVC configurations utilizing Ghanimah Block Storage NVMe disks)
|
||||||
|
|
||||||
|
### Files
|
||||||
|
- [`q4-sacalability/calculations.md`](./q4-sacalability/calculations.md)
|
||||||
|
- [`q4-sacalability/architecture.png`](./q4-sacalability/architecture.png)
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 🖥️ Question 5 – Mithal Monitoring Dashboard
|
||||||
|
|
||||||
|
### Requirements Covered
|
||||||
|
- [x] **Latency Monitoring:** Measures and logs home page loading speeds.
|
||||||
|
- [x] **Uptime Monitoring:** Tracks availability and monitors return status codes.
|
||||||
|
- [x] **SSL Monitoring:** Calculates certificate expiration and counts remaining days.
|
||||||
|
- [x] **DNS Monitoring:** Resolves and logs lookup times.
|
||||||
|
- [x] **Search Response Monitoring:** Assesses specific query parameters response time.
|
||||||
|
- [x] **JSON Storage:** Logs state to dynamic history files up to 24 hours.
|
||||||
|
- [x] **Dashboard:** An interface displaying key charts and status panels.
|
||||||
|
- [x] **Deployment on Ghaymah:** Live monitor configuration on Ghaymah infrastructure.
|
||||||
|
|
||||||
|
### Files
|
||||||
|
- [`q5-mithal-monitor/monitor.py`](file:///home/ahmed/ghaymah_task/ghaymah-exam-ahmed-abdelaziz-SRE/q5-mithal-monitor/monitor.py)
|
||||||
|
- [`q5-mithal-monitor/metrics.json`](file:///home/ahmed/ghaymah_task/ghaymah-exam-ahmed-abdelaziz-SRE/q5-mithal-monitor/metrics.json)
|
||||||
|
- [`q5-mithal-monitor/dashbourd.html`](file:///home/ahmed/ghaymah_task/ghaymah-exam-ahmed-abdelaziz-SRE/q5-mithal-monitor/dashbourd.html)
|
||||||
|
|
||||||
|
### Proof of Implementation
|
||||||
|
1. **monitor.py Running**
|
||||||
|

|
||||||
|
|
||||||
|
2. **Dashboard**
|
||||||
|

|
||||||
|
---
|
||||||
|
|
||||||
|
## 🏃 Running Locally
|
||||||
|
|
||||||
|
### Question 1 – Go API & Monitoring Agent
|
||||||
|
To test the Go Fiber API and the monitoring scripts locally:
|
||||||
|
|
||||||
|
1. **Build and Run the Go API via Docker:**
|
||||||
|
```bash
|
||||||
|
# Build the container image
|
||||||
|
docker build -t ghaymah-sre-api ./q1-deploy-monitor
|
||||||
|
|
||||||
|
# Run the container exposing port 8080
|
||||||
|
docker run -d -p 8080:8080 --name ghanimah-api ghaymah-sre-api
|
||||||
|
```
|
||||||
|
2. **Run the Bash Monitoring Agent:**
|
||||||
|
```bash
|
||||||
|
# Point the agent to the locally running Docker container
|
||||||
|
API_URL=http://localhost:8080 bash q1-deploy-monitor/health-check.sh
|
||||||
|
```
|
||||||
|
3. **View the Dashboard:**
|
||||||
|
Open [`q1-deploy-monitor/dashboard.html`](./q1-deploy-monitor/dashboard.html) directly in a web browser of your choice.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### Question 5 – Mithal Automated Monitor
|
||||||
|
To test the Python monitoring daemon locally:
|
||||||
|
|
||||||
|
1. **Install Dependencies:**
|
||||||
|
```bash
|
||||||
|
pip install requests
|
||||||
|
```
|
||||||
|
2. **Run the Monitor Script:**
|
||||||
|
```bash
|
||||||
|
python3 q5-mithal-monitor/monitor.py
|
||||||
|
```
|
||||||
|
3. **View the Mithal Dashboard:**
|
||||||
|
Open [`q5-mithal-monitor/dashbourd.html`](./q5-mithal-monitor/dashbourd.html) in your browser to view historical metrics.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## ✨ Project Highlights
|
||||||
|
|
||||||
|
This repository demonstrates complete implementation of fundamental Site Reliability Engineering practices:
|
||||||
|
- **Containerization:** Clean multi-stage lightweight builds (Go statically-linked binary in a `scratch` container).
|
||||||
|
- **Monitoring & Observability:** Real-time metrics gathering, system state logging, alerts modeling, and custom front-end status dashboards.
|
||||||
|
- **CI/CD:** Automated builds, environments targeting, manual approvals, and deployment orchestrations.
|
||||||
|
- **Incident Analysis:** Professional blameless postmortem report detailing timelines, root cause analysis, action items, auto-scaling thresholds, and early discovery strategies.
|
||||||
|
- **Scalability:** Quantitative capacity sizing for high-traffic environments (15,000 req/s), Cold Start tuning, and stateful volume management.
|
||||||
|
- **Cloud Deployment:** Orchestration using the Ghaymah Cloud platforms.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## ✍️ Author
|
||||||
|
|
||||||
|
**Ahmed Abdelaziz Hussein**
|
||||||
|
*Information Systems*
|
||||||
|
*Faculty of Computers and Information*
|
||||||
|
*Qabilah Profile:* [ahmed-abdelaziz-89943a271](https://qabilah.com/profile/ahmed-abdelaziz-89943a271/posts)
|
||||||
1
common-qabilah/qabilah-profile.txt
Normal file
@@ -0,0 +1 @@
|
|||||||
|
https://qabilah.com/profile/ahmed-abdelaziz-89943a271/posts
|
||||||
ثنائية
docs/screenshots/app-running.png
Normal file
|
بعد العرض: | الارتفاع: | الحجم: 260 KiB |
ثنائية
docs/screenshots/dashboard.png
Normal file
|
بعد العرض: | الارتفاع: | الحجم: 193 KiB |
ثنائية
docs/screenshots/manual-approval.png
Normal file
|
بعد العرض: | الارتفاع: | الحجم: 50 KiB |
ثنائية
docs/screenshots/mithal-dashboard.png
Normal file
|
بعد العرض: | الارتفاع: | الحجم: 78 KiB |
ثنائية
docs/screenshots/monitor-running.png
Normal file
|
بعد العرض: | الارتفاع: | الحجم: 31 KiB |
ثنائية
docs/screenshots/monitoring-script.png
Normal file
|
بعد العرض: | الارتفاع: | الحجم: 28 KiB |
ثنائية
docs/screenshots/staging-deployment.png
Normal file
|
بعد العرض: | الارتفاع: | الحجم: 38 KiB |
32
q1-deploy-monitor/Dockerfile
Normal file
@@ -0,0 +1,32 @@
|
|||||||
|
# Stage 1: Build binary using lightweight Go Alpine image
|
||||||
|
FROM golang:1.24-alpine AS builder
|
||||||
|
|
||||||
|
WORKDIR /app
|
||||||
|
|
||||||
|
# Cache dependencies
|
||||||
|
COPY go.mod go.sum ./
|
||||||
|
RUN go mod download
|
||||||
|
|
||||||
|
# Copy source code
|
||||||
|
COPY . .
|
||||||
|
|
||||||
|
# Build lightweight, statically linked binary stripped of debug information (-s -w)
|
||||||
|
RUN CGO_ENABLED=0 GOOS=linux go build -ldflags="-s -w" -o server main.go
|
||||||
|
|
||||||
|
# Stage 2: Minimal runtime image using scratch (~10-15MB final image size)
|
||||||
|
FROM scratch
|
||||||
|
|
||||||
|
# Copy CA certificates for HTTPS requests if needed
|
||||||
|
COPY --from=builder /etc/ssl/certs/ca-certificates.crt /etc/ssl/certs/
|
||||||
|
|
||||||
|
# Copy static binary from builder stage
|
||||||
|
COPY --from=builder /app/server /server
|
||||||
|
|
||||||
|
# Run as non-root user (nobody) for enhanced security
|
||||||
|
USER 65534:65534
|
||||||
|
|
||||||
|
# Expose port 8080
|
||||||
|
EXPOSE 8080
|
||||||
|
|
||||||
|
# Launch server
|
||||||
|
ENTRYPOINT ["/server"]
|
||||||
411
q1-deploy-monitor/dashboard.html
Normal file
@@ -0,0 +1,411 @@
|
|||||||
|
<!DOCTYPE html>
|
||||||
|
<html lang="ar" dir="rtl">
|
||||||
|
<head>
|
||||||
|
<meta charset="UTF-8">
|
||||||
|
<meta name="viewport" content="width=device-width, initial-scale=1.0">
|
||||||
|
<title>Ghaymah SRE - لوحة المراقبة</title>
|
||||||
|
<link rel="preconnect" href="https://fonts.googleapis.com">
|
||||||
|
<link rel="preconnect" href="https://fonts.gstatic.com" crossorigin>
|
||||||
|
<link href="https://fonts.googleapis.com/css2?family=Outfit:wght@300;400;600;700&family=Tajawal:wght@400;500;700;800&display=swap" rel="stylesheet">
|
||||||
|
<style>
|
||||||
|
:root {
|
||||||
|
--bg-dark: #0b0f19;
|
||||||
|
--card-bg: rgba(18, 26, 44, 0.75);
|
||||||
|
--card-border: rgba(255, 255, 255, 0.08);
|
||||||
|
--primary: #3b82f6;
|
||||||
|
--primary-glow: rgba(59, 130, 246, 0.4);
|
||||||
|
--success: #10b981;
|
||||||
|
--success-glow: rgba(16, 185, 129, 0.4);
|
||||||
|
--danger: #ef4444;
|
||||||
|
--danger-glow: rgba(239, 68, 68, 0.4);
|
||||||
|
--text-main: #f3f4f6;
|
||||||
|
--text-muted: #9ca3af;
|
||||||
|
}
|
||||||
|
|
||||||
|
* {
|
||||||
|
box-sizing: border-box;
|
||||||
|
margin: 0;
|
||||||
|
padding: 0;
|
||||||
|
}
|
||||||
|
|
||||||
|
body {
|
||||||
|
font-family: 'Tajawal', 'Outfit', sans-serif;
|
||||||
|
background-color: var(--bg-dark);
|
||||||
|
background-image:
|
||||||
|
radial-gradient(circle at 15% 20%, rgba(59, 130, 246, 0.12) 0%, transparent 40%),
|
||||||
|
radial-gradient(circle at 85% 80%, rgba(16, 185, 129, 0.08) 0%, transparent 40%);
|
||||||
|
color: var(--text-main);
|
||||||
|
min-height: 100vh;
|
||||||
|
padding: 2rem;
|
||||||
|
display: flex;
|
||||||
|
flex-direction: column;
|
||||||
|
align-items: center;
|
||||||
|
}
|
||||||
|
|
||||||
|
.container {
|
||||||
|
width: 100%;
|
||||||
|
max-width: 1100px;
|
||||||
|
}
|
||||||
|
|
||||||
|
header {
|
||||||
|
display: flex;
|
||||||
|
justify-content: space-between;
|
||||||
|
align-items: center;
|
||||||
|
margin-bottom: 2.5rem;
|
||||||
|
padding-bottom: 1.5rem;
|
||||||
|
border-bottom: 1px solid var(--card-border);
|
||||||
|
}
|
||||||
|
|
||||||
|
.title-group h1 {
|
||||||
|
font-size: 1.8rem;
|
||||||
|
font-weight: 800;
|
||||||
|
background: linear-gradient(135deg, #ffffff 0%, #9ca3af 100%);
|
||||||
|
-webkit-background-clip: text;
|
||||||
|
-webkit-text-fill-color: transparent;
|
||||||
|
display: flex;
|
||||||
|
align-items: center;
|
||||||
|
gap: 0.75rem;
|
||||||
|
}
|
||||||
|
|
||||||
|
.title-group p {
|
||||||
|
color: var(--text-muted);
|
||||||
|
font-size: 0.95rem;
|
||||||
|
margin-top: 0.3rem;
|
||||||
|
}
|
||||||
|
|
||||||
|
.live-badge {
|
||||||
|
display: flex;
|
||||||
|
align-items: center;
|
||||||
|
gap: 0.5rem;
|
||||||
|
background: rgba(16, 185, 129, 0.1);
|
||||||
|
border: 1px solid rgba(16, 185, 129, 0.3);
|
||||||
|
padding: 0.4rem 0.9rem;
|
||||||
|
border-radius: 9999px;
|
||||||
|
font-size: 0.85rem;
|
||||||
|
color: var(--success);
|
||||||
|
font-weight: 600;
|
||||||
|
}
|
||||||
|
|
||||||
|
.pulse-dot {
|
||||||
|
width: 8px;
|
||||||
|
height: 8px;
|
||||||
|
background-color: var(--success);
|
||||||
|
border-radius: 50%;
|
||||||
|
box-shadow: 0 0 10px var(--success);
|
||||||
|
animation: pulse 1.8s infinite;
|
||||||
|
}
|
||||||
|
|
||||||
|
@keyframes pulse {
|
||||||
|
0% { transform: scale(0.95); box-shadow: 0 0 0 0 rgba(16, 185, 129, 0.7); }
|
||||||
|
70% { transform: scale(1); box-shadow: 0 0 0 10px rgba(16, 185, 129, 0); }
|
||||||
|
100% { transform: scale(0.95); box-shadow: 0 0 0 0 rgba(16, 185, 129, 0); }
|
||||||
|
}
|
||||||
|
|
||||||
|
/* Grid Layout */
|
||||||
|
.metrics-grid {
|
||||||
|
display: grid;
|
||||||
|
grid-template-columns: repeat(auto-fit, minmax(280px, 1fr));
|
||||||
|
gap: 1.5rem;
|
||||||
|
margin-bottom: 2.5rem;
|
||||||
|
}
|
||||||
|
|
||||||
|
.card {
|
||||||
|
background: var(--card-bg);
|
||||||
|
backdrop-filter: blur(12px);
|
||||||
|
-webkit-backdrop-filter: blur(12px);
|
||||||
|
border: 1px solid var(--card-border);
|
||||||
|
border-radius: 1.25rem;
|
||||||
|
padding: 1.5rem;
|
||||||
|
box-shadow: 0 8px 32px 0 rgba(0, 0, 0, 0.37);
|
||||||
|
transition: transform 0.3s ease, border-color 0.3s ease;
|
||||||
|
}
|
||||||
|
|
||||||
|
.card:hover {
|
||||||
|
transform: translateY(-4px);
|
||||||
|
border-color: rgba(255, 255, 255, 0.2);
|
||||||
|
}
|
||||||
|
|
||||||
|
.card-header {
|
||||||
|
display: flex;
|
||||||
|
justify-content: space-between;
|
||||||
|
align-items: center;
|
||||||
|
color: var(--text-muted);
|
||||||
|
font-size: 0.9rem;
|
||||||
|
font-weight: 600;
|
||||||
|
margin-bottom: 1rem;
|
||||||
|
}
|
||||||
|
|
||||||
|
.card-icon {
|
||||||
|
width: 36px;
|
||||||
|
height: 36px;
|
||||||
|
border-radius: 0.6rem;
|
||||||
|
display: flex;
|
||||||
|
align-items: center;
|
||||||
|
justify-content: center;
|
||||||
|
font-size: 1.2rem;
|
||||||
|
background: rgba(255, 255, 255, 0.05);
|
||||||
|
}
|
||||||
|
|
||||||
|
.card-value {
|
||||||
|
font-size: 2.2rem;
|
||||||
|
font-weight: 700;
|
||||||
|
color: #ffffff;
|
||||||
|
font-family: 'Outfit', sans-serif;
|
||||||
|
letter-spacing: -0.5px;
|
||||||
|
}
|
||||||
|
|
||||||
|
.card-subtext {
|
||||||
|
font-size: 0.85rem;
|
||||||
|
color: var(--text-muted);
|
||||||
|
margin-top: 0.5rem;
|
||||||
|
display: flex;
|
||||||
|
align-items: center;
|
||||||
|
gap: 0.3rem;
|
||||||
|
}
|
||||||
|
|
||||||
|
/* Status Badge */
|
||||||
|
.status-tag {
|
||||||
|
display: inline-flex;
|
||||||
|
align-items: center;
|
||||||
|
gap: 0.5rem;
|
||||||
|
padding: 0.35rem 0.9rem;
|
||||||
|
border-radius: 0.6rem;
|
||||||
|
font-weight: 700;
|
||||||
|
font-size: 1.1rem;
|
||||||
|
}
|
||||||
|
|
||||||
|
.status-healthy {
|
||||||
|
background: rgba(16, 185, 129, 0.15);
|
||||||
|
color: var(--success);
|
||||||
|
border: 1px solid rgba(16, 185, 129, 0.4);
|
||||||
|
box-shadow: 0 0 15px var(--success-glow);
|
||||||
|
}
|
||||||
|
|
||||||
|
.status-down {
|
||||||
|
background: rgba(239, 68, 68, 0.15);
|
||||||
|
color: var(--danger);
|
||||||
|
border: 1px solid rgba(239, 68, 68, 0.4);
|
||||||
|
box-shadow: 0 0 15px var(--danger-glow);
|
||||||
|
}
|
||||||
|
|
||||||
|
/* Activity Table */
|
||||||
|
.logs-section {
|
||||||
|
background: var(--card-bg);
|
||||||
|
border: 1px solid var(--card-border);
|
||||||
|
border-radius: 1.25rem;
|
||||||
|
padding: 1.5rem;
|
||||||
|
backdrop-filter: blur(12px);
|
||||||
|
}
|
||||||
|
|
||||||
|
.logs-header {
|
||||||
|
display: flex;
|
||||||
|
justify-content: space-between;
|
||||||
|
align-items: center;
|
||||||
|
margin-bottom: 1rem;
|
||||||
|
}
|
||||||
|
|
||||||
|
.logs-table {
|
||||||
|
width: 100%;
|
||||||
|
border-collapse: collapse;
|
||||||
|
text-align: right;
|
||||||
|
font-size: 0.9rem;
|
||||||
|
}
|
||||||
|
|
||||||
|
.logs-table th {
|
||||||
|
color: var(--text-muted);
|
||||||
|
font-weight: 600;
|
||||||
|
padding: 0.75rem 1rem;
|
||||||
|
border-bottom: 1px solid var(--card-border);
|
||||||
|
}
|
||||||
|
|
||||||
|
.logs-table td {
|
||||||
|
padding: 0.85rem 1rem;
|
||||||
|
border-bottom: 1px solid rgba(255, 255, 255, 0.04);
|
||||||
|
color: var(--text-main);
|
||||||
|
font-family: 'Outfit', 'Tajawal', sans-serif;
|
||||||
|
}
|
||||||
|
|
||||||
|
.badge-pill {
|
||||||
|
padding: 0.2rem 0.6rem;
|
||||||
|
border-radius: 0.4rem;
|
||||||
|
font-size: 0.75rem;
|
||||||
|
font-weight: 600;
|
||||||
|
}
|
||||||
|
|
||||||
|
.pill-200 { background: rgba(16, 185, 129, 0.2); color: var(--success); }
|
||||||
|
.pill-err { background: rgba(239, 68, 68, 0.2); color: var(--danger); }
|
||||||
|
</style>
|
||||||
|
</head>
|
||||||
|
<body>
|
||||||
|
<div class="container">
|
||||||
|
<!-- Header -->
|
||||||
|
<header>
|
||||||
|
<div class="title-group">
|
||||||
|
<h1>🌩️ Ghaymah SRE Monitor</h1>
|
||||||
|
<p>مراقبة أداء واستجابة التطبيق المباشرة (Real-time Health Dashboard)</p>
|
||||||
|
</div>
|
||||||
|
<div class="live-badge">
|
||||||
|
<span class="pulse-dot"></span>
|
||||||
|
<span id="lastUpdatedText">مباشر (تحديث كل 3 ثوانٍ)</span>
|
||||||
|
</div>
|
||||||
|
</header>
|
||||||
|
|
||||||
|
<!-- Metrics Cards -->
|
||||||
|
<div class="metrics-grid">
|
||||||
|
<!-- 1. Status Card -->
|
||||||
|
<div class="card">
|
||||||
|
<div class="card-header">
|
||||||
|
<span>حالة النظام (Status)</span>
|
||||||
|
<div class="card-icon">⚡</div>
|
||||||
|
</div>
|
||||||
|
<div id="statusBadgeContainer">
|
||||||
|
<span id="statusTag" class="status-tag status-healthy">● متصل (Healthy)</span>
|
||||||
|
</div>
|
||||||
|
<div class="card-subtext" id="httpCodeSubtext">كود الاستجابة: HTTP 200 OK</div>
|
||||||
|
</div>
|
||||||
|
|
||||||
|
<!-- 2. Latency Card -->
|
||||||
|
<div class="card">
|
||||||
|
<div class="card-header">
|
||||||
|
<span>زمن الاستجابة (Latency)</span>
|
||||||
|
<div class="card-icon">⏱️</div>
|
||||||
|
</div>
|
||||||
|
<div class="card-value"><span id="latencyValue">--</span> <small style="font-size: 1rem; font-weight: 400; color: var(--text-muted);">ms</small></div>
|
||||||
|
<div class="card-subtext">
|
||||||
|
<span>متوسط الاستجابة:</span>
|
||||||
|
<strong id="avgLatencyValue" style="color: var(--primary);">-- ms</strong>
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
|
||||||
|
<!-- 3. Total Requests Card -->
|
||||||
|
<div class="card">
|
||||||
|
<div class="card-header">
|
||||||
|
<span>إجمالي الطلبات (Total Requests)</span>
|
||||||
|
<div class="card-icon">📊</div>
|
||||||
|
</div>
|
||||||
|
<div class="card-value" id="totalRequestsValue">--</div>
|
||||||
|
<div class="card-subtext">
|
||||||
|
<span>مدة التشغيل (Uptime):</span>
|
||||||
|
<strong id="uptimeValue" style="color: #ffffff;">-- ثانية</strong>
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
|
||||||
|
<!-- Check Logs -->
|
||||||
|
<div class="logs-section">
|
||||||
|
<div class="logs-header">
|
||||||
|
<h3>سجل الفحوصات الأخيرة (Health Check Logs)</h3>
|
||||||
|
<span style="font-size: 0.85rem; color: var(--text-muted);" id="apiTargetUrl">الهدف: http://localhost:8080/metrics</span>
|
||||||
|
</div>
|
||||||
|
<table class="logs-table">
|
||||||
|
<thead>
|
||||||
|
<tr>
|
||||||
|
<th>الوقت (UTC)</th>
|
||||||
|
<th>الحالة</th>
|
||||||
|
<th>كود HTTP</th>
|
||||||
|
<th>زمن الاستجابة</th>
|
||||||
|
<th>إجمالي الطلبات</th>
|
||||||
|
</tr>
|
||||||
|
</thead>
|
||||||
|
<tbody id="logsTableBody">
|
||||||
|
<tr>
|
||||||
|
<td colspan="5" style="text-align: center; color: var(--text-muted); padding: 1.5rem;">جاري جلب البيانات...</td>
|
||||||
|
</tr>
|
||||||
|
</tbody>
|
||||||
|
</table>
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
|
||||||
|
<script>
|
||||||
|
const API_METRICS_URL = 'http://localhost:8080/metrics';
|
||||||
|
const FALLBACK_STATUS_URL = 'status.json';
|
||||||
|
const logsHistory = [];
|
||||||
|
|
||||||
|
async function fetchMetrics() {
|
||||||
|
let data = null;
|
||||||
|
let source = 'API';
|
||||||
|
|
||||||
|
try {
|
||||||
|
// Try fetching direct API endpoint
|
||||||
|
const response = await fetch(API_METRICS_URL, { cache: 'no-store' });
|
||||||
|
if (response.ok) {
|
||||||
|
data = await response.json();
|
||||||
|
}
|
||||||
|
} catch (err) {
|
||||||
|
// Fallback to status.json if direct API fails or CORS blocked
|
||||||
|
try {
|
||||||
|
const statusRes = await fetch(FALLBACK_STATUS_URL, { cache: 'no-store' });
|
||||||
|
if (statusRes.ok) {
|
||||||
|
data = await statusRes.json();
|
||||||
|
source = 'status.json';
|
||||||
|
}
|
||||||
|
} catch (e) {
|
||||||
|
data = null;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
updateDashboardUI(data);
|
||||||
|
}
|
||||||
|
|
||||||
|
function updateDashboardUI(data) {
|
||||||
|
const statusTag = document.getElementById('statusTag');
|
||||||
|
const httpCodeSubtext = document.getElementById('httpCodeSubtext');
|
||||||
|
const latencyValue = document.getElementById('latencyValue');
|
||||||
|
const avgLatencyValue = document.getElementById('avgLatencyValue');
|
||||||
|
const totalRequestsValue = document.getElementById('totalRequestsValue');
|
||||||
|
const uptimeValue = document.getElementById('uptimeValue');
|
||||||
|
|
||||||
|
const nowIso = new Date().toLocaleTimeString('ar-SA');
|
||||||
|
|
||||||
|
if (data && (data.status === 'healthy' || data.status === 'ok')) {
|
||||||
|
statusTag.className = 'status-tag status-healthy';
|
||||||
|
statusTag.innerHTML = '● متصل (Healthy)';
|
||||||
|
httpCodeSubtext.textContent = 'كود الاستجابة: HTTP 200 OK';
|
||||||
|
|
||||||
|
const latency = data.last_request_latency_ms || data.check_latency_ms || 0;
|
||||||
|
latencyValue.textContent = latency;
|
||||||
|
avgLatencyValue.textContent = (data.average_latency_ms || 0) + ' ms';
|
||||||
|
totalRequestsValue.textContent = (data.total_requests || 0).toLocaleString();
|
||||||
|
uptimeValue.textContent = (data.uptime_seconds || 0) + ' ثانية';
|
||||||
|
|
||||||
|
addLogEntry(nowIso, 'Healthy', 200, latency + ' ms', data.total_requests || 0);
|
||||||
|
} else {
|
||||||
|
statusTag.className = 'status-tag status-down';
|
||||||
|
statusTag.innerHTML = '✖ غير متصل (Down)';
|
||||||
|
httpCodeSubtext.textContent = 'كود الاستجابة: HTTP 500 / Connection Error';
|
||||||
|
|
||||||
|
latencyValue.textContent = '--';
|
||||||
|
avgLatencyValue.textContent = '-- ms';
|
||||||
|
totalRequestsValue.textContent = '--';
|
||||||
|
uptimeValue.textContent = '0 ثانية';
|
||||||
|
|
||||||
|
addLogEntry(nowIso, 'Down', 500, '--', '--');
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
function addLogEntry(time, status, code, latency, reqCount) {
|
||||||
|
logsHistory.unshift({ time, status, code, latency, reqCount });
|
||||||
|
if (logsHistory.length > 7) logsHistory.pop();
|
||||||
|
|
||||||
|
const tbody = document.getElementById('logsTableBody');
|
||||||
|
tbody.innerHTML = logsHistory.map(entry => `
|
||||||
|
<tr>
|
||||||
|
<td>${entry.time}</td>
|
||||||
|
<td>
|
||||||
|
<span class="badge-pill ${entry.status === 'Healthy' ? 'pill-200' : 'pill-err'}">
|
||||||
|
${entry.status}
|
||||||
|
</span>
|
||||||
|
</td>
|
||||||
|
<td>HTTP ${entry.code}</td>
|
||||||
|
<td>${entry.latency}</td>
|
||||||
|
<td>${entry.reqCount}</td>
|
||||||
|
</tr>
|
||||||
|
`).join('');
|
||||||
|
}
|
||||||
|
|
||||||
|
// Initial fetch & set interval (3 seconds)
|
||||||
|
fetchMetrics();
|
||||||
|
setInterval(fetchMetrics, 3000);
|
||||||
|
</script>
|
||||||
|
</body>
|
||||||
|
</html>
|
||||||
68
q1-deploy-monitor/health-check.sh
Executable file
@@ -0,0 +1,68 @@
|
|||||||
|
#!/bin/bash
|
||||||
|
|
||||||
|
# Configuration
|
||||||
|
API_URL="${API_URL:-https://ghaymah-task-da46b224b9fc.hosted.ghaymah.systems}"
|
||||||
|
CHECK_INTERVAL=30
|
||||||
|
STATUS_FILE="status.json"
|
||||||
|
|
||||||
|
echo "=================================================="
|
||||||
|
echo " Starting Ghaymah SRE Monitoring Agent "
|
||||||
|
echo " Target URL : $API_URL "
|
||||||
|
echo " Interval : ${CHECK_INTERVAL}s "
|
||||||
|
echo "=================================================="
|
||||||
|
|
||||||
|
while true; do
|
||||||
|
TIMESTAMP=$(date -u +"%Y-%m-%dT%H:%M:%SZ")
|
||||||
|
|
||||||
|
START_TIME=$(date +%s%N)
|
||||||
|
HTTP_RESPONSE=$(curl -s -w "\n%{http_code}" "${API_URL}/metrics")
|
||||||
|
END_TIME=$(date +%s%N)
|
||||||
|
|
||||||
|
# Calculate latency in milliseconds
|
||||||
|
LATENCY_MS=$(( (END_TIME - START_TIME) / 1000000 ))
|
||||||
|
|
||||||
|
HTTP_BODY=$(echo "$HTTP_RESPONSE" | sed '$d')
|
||||||
|
HTTP_CODE=$(echo "$HTTP_RESPONSE" | tail -n1)
|
||||||
|
|
||||||
|
if [ "$HTTP_CODE" -eq 200 ]; then
|
||||||
|
STATUS="healthy"
|
||||||
|
UPTIME=$(echo "$HTTP_BODY" | grep -o '"uptime_seconds":[0-9]*' | cut -d':' -f2)
|
||||||
|
TOTAL_REQ=$(echo "$HTTP_BODY" | grep -o '"total_requests":[0-9]*' | cut -d':' -f2)
|
||||||
|
AVG_LATENCY=$(echo "$HTTP_BODY" | grep -o '"average_latency_ms":[0-9.]*' | cut -d':' -f2)
|
||||||
|
LAST_LATENCY=$(echo "$HTTP_BODY" | grep -o '"last_request_latency_ms":[0-9.]*' | cut -d':' -f2)
|
||||||
|
|
||||||
|
echo "[ $TIMESTAMP ] [OK 200] Status: $STATUS | Latency: ${LATENCY_MS}ms | Total Requests: ${TOTAL_REQ:-0} | Avg Latency: ${AVG_LATENCY:-0}ms"
|
||||||
|
|
||||||
|
# Export JSON metrics status file for dashboard consumption
|
||||||
|
cat <<EOF > "$STATUS_FILE"
|
||||||
|
{
|
||||||
|
"status": "$STATUS",
|
||||||
|
"http_code": $HTTP_CODE,
|
||||||
|
"check_timestamp": "$TIMESTAMP",
|
||||||
|
"check_latency_ms": $LATENCY_MS,
|
||||||
|
"uptime_seconds": ${UPTIME:-0},
|
||||||
|
"total_requests": ${TOTAL_REQ:-0},
|
||||||
|
"average_latency_ms": ${AVG_LATENCY:-0},
|
||||||
|
"last_request_latency_ms": ${LAST_LATENCY:-$LATENCY_MS}
|
||||||
|
}
|
||||||
|
EOF
|
||||||
|
else
|
||||||
|
STATUS="unhealthy"
|
||||||
|
echo "[ $TIMESTAMP ] [ERROR ${HTTP_CODE:-000}] Target $API_URL is DOWN or unreachable! Latency: ${LATENCY_MS}ms"
|
||||||
|
|
||||||
|
cat <<EOF > "$STATUS_FILE"
|
||||||
|
{
|
||||||
|
"status": "$STATUS",
|
||||||
|
"http_code": ${HTTP_CODE:-0},
|
||||||
|
"check_timestamp": "$TIMESTAMP",
|
||||||
|
"check_latency_ms": $LATENCY_MS,
|
||||||
|
"uptime_seconds": 0,
|
||||||
|
"total_requests": 0,
|
||||||
|
"average_latency_ms": 0,
|
||||||
|
"last_request_latency_ms": 0
|
||||||
|
}
|
||||||
|
EOF
|
||||||
|
fi
|
||||||
|
|
||||||
|
sleep $CHECK_INTERVAL
|
||||||
|
done
|
||||||
10
q1-deploy-monitor/status.json
Normal file
@@ -0,0 +1,10 @@
|
|||||||
|
{
|
||||||
|
"status": "healthy",
|
||||||
|
"http_code": 200,
|
||||||
|
"check_timestamp": "2026-07-28T10:09:20Z",
|
||||||
|
"check_latency_ms": 67,
|
||||||
|
"uptime_seconds": 5973,
|
||||||
|
"total_requests": 203,
|
||||||
|
"average_latency_ms": 1.76,
|
||||||
|
"last_request_latency_ms": 0.27
|
||||||
|
}
|
||||||
146
q2-postmortem/postmortem-report.md
Normal file
@@ -0,0 +1,146 @@
|
|||||||
|
# 📋 تقرير تحليل حادثة انقطاع الخدمة — Postmortem Report
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 1️⃣ تقرير الـ Postmortem التفصيلي
|
||||||
|
|
||||||
|
### 📑 الملخص التنفيذي (Executive Summary)
|
||||||
|
* **اسم الحادثة:** INC-2026-0728-OOM
|
||||||
|
* **تاريخ الحادثة:** 28 يوليو 2026
|
||||||
|
* **مدة الانقطاع (Downtime):** 45 دقيقة (10:00 - 10:45 UTC)
|
||||||
|
* **مستوى الأهمية (Severity):** Critical (SEV-1)
|
||||||
|
* **السبب الرئيسي:** حدوث حالات `OOMKilled` (Out Of Memory) متكررة للحاويات نتيجة ارتفاع مفاجئ في حركة المرور (Traffic Spike) مع عدم وجود سياسة للتوسع التلقائي (Auto-scaling) وضيق حدود الذاكرة المخصصة.
|
||||||
|
* **الأثر على الخدمة:** توقف تام لأحد الخدمات الأساسية (HTTP 502 Bad Gateway) وتضرر 100% من طلبات المستخدمين أثناء فترة الانقطاع.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### ⏱️ الجدول الزمني للحادثة (Timeline of Events)
|
||||||
|
|
||||||
|
| الوقت (UTC) | الحدث |
|
||||||
|
| :--- | :--- |
|
||||||
|
| **10:00** | بدء ارتفاع مفاجئ في عدد الطلبات الموجهة للخدمة (تضاعف عدد الطلبات 5 مرات). |
|
||||||
|
| **10:05** | تجاوز استهلاك الذاكرة الحد الأقصى المخصص (`Memory Limit = 512MiB`)، وقام الـ Linux Kernel بإنهاء الحاوية (`OOMKilled - Exit Code 137`). |
|
||||||
|
| **10:08** | دخول الحاوية في حالة `CrashLoopBackOff` بسبب تكرار الـ OOM فور إعادة التشغيل. |
|
||||||
|
| **10:12** | انطلاق أول تنبيه (Health Check Failure) وفشلت فحوصات الجاهزية (Liveness & Readiness Probes). |
|
||||||
|
| **10:20** | استجابة فريق الـ SRE وبدء التحقيق في المشكلة عبر مراجعة السجلات والتنبيهات. |
|
||||||
|
| **10:30** | تحديد السبب الجذر: حدوث `OOMKilled` متكرر لجميع الـ Replicas الشغالة. |
|
||||||
|
| **10:38** | **إجراء طارئ:** رفع حدود الذاكرة يدويًا من `512MiB` إلى `2GiB` وزيادة عدد الـ Pods من 2 إلى 6. |
|
||||||
|
| **10:45** | استقرار جميع الحاويات، وعودة مؤشرات الخدمة إلى الوضع الطبيعي (HTTP 200 OK) وانتهاء الحادثة. |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### 🔍 السبب الجذر (Root Cause Analysis - RCA)
|
||||||
|
|
||||||
|
1. **قصور في تخصيص الموارد (Insufficient Memory Limits):**
|
||||||
|
* تم تخصيص حد ذاكرة منخفض جِدًّا (`512MiB`) لا يتناسب مع حجم العمل المرتفع، مما جعل التطبيق عرضة للإنهاء المباشر بواسطة الـ OOM Killer عند حدوث ضغط.
|
||||||
|
2. **غياب التوسع التلقائي (No Auto-scaling):**
|
||||||
|
* لم يتم إعداد `HorizontalPodAutoscaler` (HPA) للتوسع الأفقي عند ارتفاع الضغط.
|
||||||
|
3. **تنبيهات متأخرة للذاكرة:**
|
||||||
|
* التنبيهات كانت مجهزة فقط عند توقف الخدمة تمامًا (`Health Check Failed`) بدلاً من التنبيه المبكر عند وصول استهلاك الذاكرة إلى 80%.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### 🛠️ التوصيات والإجراءات التصحيحية (Action Items)
|
||||||
|
|
||||||
|
| الرقم | الإجراء (Action Item) | النوع | الأولوية |
|
||||||
|
| :---: | :--- | :---: | :---: |
|
||||||
|
| **1** | زيادة الـ Memory Limits الأساسية من `512MiB` إلى `1GiB` وتحديد `Requests` عند `512MiB`. | فوري (Immediate) | 🔴 P0 |
|
||||||
|
| **2** | تطبيق سياسة Auto-scaling أفقية (HPA) لمنصة غنيمة تعمل على الذاكرة والـ CPU. | متوسط (Medium) | 🔴 P0 |
|
||||||
|
| **3** | إضافة قواعد تنبيه مبكرة عند تجاوز الذاكرة نسبة 80% وقبل الوصول لـ 100%. | فوري (Immediate) | 🟡 P1 |
|
||||||
|
| **4** | إجبار إجراء اختبارات ضغط (Load & Stress Testing) قبل الترقية للإنتاج. | طويل المدى | 🟢 P2 |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 2️⃣ تصميم سياسة Auto-scaling لمنصة غنيمة (Ghanimah Auto-scaling Policy)
|
||||||
|
|
||||||
|
لتفادي تكرار حادثة `OOMKilled` مستقبلاً، تم تصميم سياسة **Horizontal Pod Autoscaler (HPA)** مخصصة لمنصة **غنيمة**:
|
||||||
|
|
||||||
|
### 📐 المكونات والحدود (Policy Configuration)
|
||||||
|
|
||||||
|
```yaml
|
||||||
|
apiVersion: autoscaling/v2
|
||||||
|
kind: HorizontalPodAutoscaler
|
||||||
|
metadata:
|
||||||
|
name: ghanimah-app-hpa
|
||||||
|
namespace: production
|
||||||
|
spec:
|
||||||
|
scaleTargetRef:
|
||||||
|
apiVersion: apps/v1
|
||||||
|
kind: Deployment
|
||||||
|
name: ghanimah-sre-api
|
||||||
|
minReplicas: 3 # الحد الأدنى لضمان العزل والتوافر العالي
|
||||||
|
maxReplicas: 15 # الحد الأقصى للاستجابة للهجمات أو الضغط العالي
|
||||||
|
metrics:
|
||||||
|
# 1. التوسع بناءً على الذاكرة (Memory Utilization) - منع الـ OOM
|
||||||
|
- type: Resource
|
||||||
|
resource:
|
||||||
|
name: memory
|
||||||
|
target:
|
||||||
|
type: Utilization
|
||||||
|
averageUtilization: 70 # التوسع فور وصول الذاكرة إلى 70%
|
||||||
|
# 2. التوسع بناءً على المعالج (CPU Utilization)
|
||||||
|
- type: Resource
|
||||||
|
resource:
|
||||||
|
name: cpu
|
||||||
|
target:
|
||||||
|
type: Utilization
|
||||||
|
averageUtilization: 75
|
||||||
|
behavior:
|
||||||
|
scaleUp:
|
||||||
|
stabilizationWindowSeconds: 0 # توسع فوري (0 ثانية) لتجنب OOMKilled
|
||||||
|
policies:
|
||||||
|
- type: Percent
|
||||||
|
value: 100 # مضاعفة عدد الـ Pods فوراً عند الضغط
|
||||||
|
periodSeconds: 15
|
||||||
|
scaleDown:
|
||||||
|
stabilizationWindowSeconds: 300 # الانتظار 5 دقائق قبل تقليص العدد لمنع Flapping
|
||||||
|
```
|
||||||
|
|
||||||
|
### 💡 قواعد الموارد في الحاوية (Resource Requests & Limits)
|
||||||
|
```yaml
|
||||||
|
resources:
|
||||||
|
requests:
|
||||||
|
cpu: "250m"
|
||||||
|
memory: "512Mi"
|
||||||
|
limits:
|
||||||
|
cpu: "1000m"
|
||||||
|
memory: "1024Mi"
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 3️⃣ الكشف المبكر والمراقبة باستخدام أدوات غنيمة (Early Detection & Observability)
|
||||||
|
|
||||||
|
لكشف مشكلة الذاكرة قبل وصولها لمرحلة `OOMKilled`:
|
||||||
|
|
||||||
|
### 1. إعداد قواعد التنبيه المبكر (Prometheus Alert Rules)
|
||||||
|
|
||||||
|
* **تنبيه تحذيري لارتفاع الذاكرة (Memory Usage Warning > 80%):**
|
||||||
|
```yaml
|
||||||
|
alert: HighMemoryUsageWarning
|
||||||
|
expr: (container_memory_working_set_bytes{container!=""} / container_spec_memory_limit_bytes{container!=""}) * 100 > 80
|
||||||
|
for: 2m
|
||||||
|
labels:
|
||||||
|
severity: warning
|
||||||
|
annotations:
|
||||||
|
summary: "استهلاك الذاكرة تجاوز 80% في الحاوية {{ $labels.pod }}"
|
||||||
|
```
|
||||||
|
|
||||||
|
* **تنبيه عاجل لحدث OOMKilled أو إعادة تشغيل متكررة:**
|
||||||
|
```yaml
|
||||||
|
alert: ContainerOOMKilledDetected
|
||||||
|
expr: increase(kube_pod_container_status_restarts_total[5m]) > 2
|
||||||
|
for: 0m
|
||||||
|
labels:
|
||||||
|
severity: critical
|
||||||
|
annotations:
|
||||||
|
summary: "تم اكتشاف إعادة تشغيل متكررة للحاوية {{ $labels.pod }} - احتمال OOMKilled"
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### 2. أهم المؤشرات في لوحة المراقبة (Dashboard Metrics)
|
||||||
|
|
||||||
|
1. **`container_memory_working_set_bytes`**: قياس حجم الذاكرة المستخدمة فعلياً مقارنة بالحد الأقصى (`container_spec_memory_limit_bytes`).
|
||||||
|
2. **`kube_pod_container_status_last_terminated_reason`**: الكشف الفوري عن قيمة `OOMKilled`.
|
||||||
|
3. **`rate(http_requests_total[1m])`**: متابعة نمو الطلبات للتنبؤ بالضغط والتوسع المبكر.
|
||||||
75
q3-cicd/workflow.yml
Normal file
@@ -0,0 +1,75 @@
|
|||||||
|
name: Ghaymah CI/CD Pipeline
|
||||||
|
|
||||||
|
on:
|
||||||
|
push:
|
||||||
|
branches:
|
||||||
|
- "release/**"
|
||||||
|
- main
|
||||||
|
workflow_dispatch:
|
||||||
|
|
||||||
|
jobs:
|
||||||
|
build:
|
||||||
|
name: Build Docker Image
|
||||||
|
runs-on: ubuntu-latest
|
||||||
|
|
||||||
|
steps:
|
||||||
|
- name: Checkout Source
|
||||||
|
uses: actions/checkout@v5
|
||||||
|
|
||||||
|
- name: Build Docker Image
|
||||||
|
run: |
|
||||||
|
docker build -t ghaymah-sre-api:${{ github.sha }} .
|
||||||
|
|
||||||
|
deploy-staging:
|
||||||
|
name: Deploy to Staging
|
||||||
|
needs: build
|
||||||
|
|
||||||
|
if: startsWith(github.ref, 'refs/heads/release/')
|
||||||
|
|
||||||
|
environment: staging
|
||||||
|
|
||||||
|
runs-on: ubuntu-latest
|
||||||
|
|
||||||
|
steps:
|
||||||
|
- uses: actions/checkout@v5
|
||||||
|
|
||||||
|
- name: Install Ghaymah CLI
|
||||||
|
run: curl -sSL https://cli.ghaymah.systems/install.sh | bash
|
||||||
|
|
||||||
|
- name: Login to Ghaymah
|
||||||
|
run: |
|
||||||
|
$HOME/ghaymah/bin/gy auth login \
|
||||||
|
--email "${{ secrets.GHAYMAH_EMAIL }}" \
|
||||||
|
--password "${{ secrets.GHAYMAH_PW }}"
|
||||||
|
|
||||||
|
- name: Deploy to Ghaymah Staging
|
||||||
|
run: |
|
||||||
|
$HOME/ghaymah/bin/gy resource app launch
|
||||||
|
|
||||||
|
deploy-production:
|
||||||
|
name: Deploy to Production
|
||||||
|
|
||||||
|
needs: build
|
||||||
|
|
||||||
|
if: github.ref == 'refs/heads/main'
|
||||||
|
|
||||||
|
environment:
|
||||||
|
name: production
|
||||||
|
|
||||||
|
runs-on: ubuntu-latest
|
||||||
|
|
||||||
|
steps:
|
||||||
|
- uses: actions/checkout@v5
|
||||||
|
|
||||||
|
- name: Install Ghaymah CLI
|
||||||
|
run: curl -sSL https://cli.ghaymah.systems/install.sh | bash
|
||||||
|
|
||||||
|
- name: Login to Ghaymah
|
||||||
|
run: |
|
||||||
|
$HOME/ghaymah/bin/gy auth login \
|
||||||
|
--email "${{ secrets.GHAYMAH_EMAIL }}" \
|
||||||
|
--password "${{ secrets.GHAYMAH_PW }}"
|
||||||
|
|
||||||
|
- name: Deploy to Ghaymah Production
|
||||||
|
run: |
|
||||||
|
$HOME/ghaymah/bin/gy resource app launch
|
||||||
ثنائية
q4-sacalability/architecture.png
Normal file
|
بعد العرض: | الارتفاع: | الحجم: 1.4 MiB |
110
q4-sacalability/calculations.md
Normal file
@@ -0,0 +1,110 @@
|
|||||||
|
# 🚀 قابلية التوسع وتوزيع الأحمال على منصة غنيمة (Scalability & Load Balancing)
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 1️⃣ المخطط الهندسي للنظام (Architecture Diagram — 15,000 req/s)
|
||||||
|
|
||||||
|
تم تصميم المعمارية التالية للاستجابة لـ **15,000 طلب في الثانية (15,000 req/s)** مع توافر عالي (High Availability) وأداء متفوق على منصة **غنيمة**:
|
||||||
|
|
||||||
|

|
||||||
|
|
||||||
|
## 2️⃣ حساب عدد الحاويات المطلوبة (Capacity Planning & Sizing)
|
||||||
|
|
||||||
|
### 📊 المعطيات:
|
||||||
|
* **حركة المرور المستهدفة (Target Traffic):** $15,000 \text{ req/s}$
|
||||||
|
* **طاقة الحاوية الواحدة (Container Capacity):** $500 \text{ req/s}$
|
||||||
|
* **هامش الأمان الموصى به (Safety Margin Buffer):** $30\%$
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### 🧮 الخطوات الحسابية:
|
||||||
|
|
||||||
|
1. **حساب إجمالي حركة المرور المطلوبة مع هامش الأمان:**
|
||||||
|
$$\text{Total Traffic with Buffer} = 15,000 \times (1 + 0.30) = 15,000 \times 1.30 = 19,500 \text{ req/s}$$
|
||||||
|
|
||||||
|
2. **حساب عدد الحاويات المطلوبة:**
|
||||||
|
$$\text{Number of Containers} = \left\lceil \frac{19,500 \text{ req/s}}{500 \text{ req/s}} \right\rceil = 39 \text{ Containers}$$
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### 📌 النتيجة والتوزيع على بيئات غنيمة:
|
||||||
|
* **إجمالي عدد الحاويات (Pods):** **39 حاوية** (تضمن معالجة $19,500 \text{ req/s}$ بكفاءة عالية وبدون اختناق).
|
||||||
|
* **توزيع الحاويات على مناطق التوافر (Multi-AZ Deployment):**
|
||||||
|
* **Zone A:** 13 Pods
|
||||||
|
* **Zone B:** 13 Pods
|
||||||
|
* **Zone C:** 13 Pods
|
||||||
|
* **سبب إضافة هامش الأمان (30% Buffer):**
|
||||||
|
1. امتصاص الارتفاعات المفاجئة واللحظية في حركة المرور (Traffic Spikes).
|
||||||
|
2. تغطية استهلاك الموارد الموجه لـ Health Checks و Garbage Collection.
|
||||||
|
3. حماية النظام أثناء إعادة تشغيل الحاويات أو تحديثات النسخ (Rolling Updates).
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 3️⃣ استراتيجية تقليل الـ Cold Start للحاويات الجديدة
|
||||||
|
|
||||||
|
الـ **Cold Start** هو الوقت المستغرق بين إطلاق حاوية جديدة وجاهزيتها التامة لاستقبال الطلبات. لتقليل هذا الوقت لأقل من 1 ثانية في منصة **غنيمة**، نتبع الاستراتيجيات التالية:
|
||||||
|
|
||||||
|
### 1. تصغير حجم صورة الحاوية (Lightweight Docker Images)
|
||||||
|
* استخدام صور **Multi-stage Build** مبنية على `scratch` أو `alpine` (حجم الصورة النهائي **~6.7 ميجابايت** كما تم بناؤه في Dockerfile غنيمة).
|
||||||
|
* الصورة الصغيرة يسهل سحبها من **Ghanimah Container Registry** عبر شبكة غنيمة السريعة خلال مسبارات زمنية تقل عن **200ms**.
|
||||||
|
|
||||||
|
### 2. التوسع الاستباقي (Proactive Pre-Warming & Buffer Capacity)
|
||||||
|
* ضبط الحد الأدنى للحاويات عند 39 حاوية، وتفعيل التوسع التلقائي (HPA) فور وصول استهلاك الذاكرة أو المعالج إلى **70%** (بدلاً من 90%).
|
||||||
|
* هذا يمنح الـ Clusters وقتاً كافياً لإطلاق الحاويات قبل وصول الضغط الفعلي للذروة.
|
||||||
|
|
||||||
|
### 3. تحسين فحوصات الجاهزية (Readiness Probes Tuning)
|
||||||
|
* ضبط فحوصات الجاهزية للبدء سريعاً بدون تأخير غير برمجيات:
|
||||||
|
```yaml
|
||||||
|
readinessProbe:
|
||||||
|
httpGet:
|
||||||
|
path: /health
|
||||||
|
port: 8080
|
||||||
|
initialDelaySeconds: 1 # البدء بالفحص فوراً بعد ثانية واحدة
|
||||||
|
periodSeconds: 2 # الفحص كل ثانيتين
|
||||||
|
successThreshold: 1
|
||||||
|
failureThreshold: 2
|
||||||
|
```
|
||||||
|
|
||||||
|
### 4. تسريع وقت تشغيل التطبيق (Fast Application Startup)
|
||||||
|
* الاعتماد على لغة Go المجمعة بلغة الآلة (Native Compiled Output) والتي تبدأ العمل خلال أجزاء من الملي ثانية بدون تجمعات JVM أو حزم تفسير ثقيلة.
|
||||||
|
* تجميع وإيقاف أي اتصالات قاعدة بيانات كسولة (Lazy Initialization) واستبدالها باتصالات جاهزة سلفاً (Pre-warmed connection pools).
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 4️⃣ استخدام Ghanimah Block Storage للبيانات المستمرة (Stateful Workloads)
|
||||||
|
|
||||||
|
تعتمد الحاويات بطبيعتها على كونها **Stateless** (تزول بياناتها بزوال الحاوية). ولتشغيل التطبيقات التي تتطلب حفظ البيانات بشكل دائم (Stateful Workloads مثل قواعد البيانات PostgreSQL و Redis Persistent Logs)، توفر منصة غنيمة **Ghanimah Block Storage (GBS)**.
|
||||||
|
|
||||||
|
### 🔑 أهم الميزات والية العمل:
|
||||||
|
|
||||||
|
```text
|
||||||
|
┌────────────────────────┐ Persistent Volume Claim ┌────────────────────────────┐
|
||||||
|
│ PostgreSQL Pod │ ───────────────────────────────────► │ Ghanimah Block Storage(PV) │
|
||||||
|
│ (Stateful Workload) │ (Attach NVMe Storage) │ (High-Performance SSD) │
|
||||||
|
└────────────────────────┘ └────────────────────────────┘
|
||||||
|
```
|
||||||
|
|
||||||
|
1. **الأداء العالي (High Performance NVMe Volumes):**
|
||||||
|
* يوفر Ghanimah Block Storage أقراص NVMe فائقة السرعة مع معدل عمليات إدخال/إخراج يصل إلى **60,000 IOPS** وتأخير أقل من **1ms**، وهو مثالي لقواعد البيانات الضخمة.
|
||||||
|
|
||||||
|
2. **التكامل عبر Kubernetes Dynamic Provisioning (PVC & StorageClass):**
|
||||||
|
* يتم ربط التخزين بالتطبيقات باستخدام `PersistentVolumeClaim` (PVC) وتحديد `StorageClass: ghanimah-block-nvme`:
|
||||||
|
```yaml
|
||||||
|
apiVersion: v1
|
||||||
|
kind: PersistentVolumeClaim
|
||||||
|
metadata:
|
||||||
|
name: ghanimah-db-pvc
|
||||||
|
spec:
|
||||||
|
accessModes:
|
||||||
|
- ReadWriteOnce
|
||||||
|
storageClassName: ghanimah-block-nvme
|
||||||
|
resources:
|
||||||
|
requests:
|
||||||
|
storage: 250Gi
|
||||||
|
```
|
||||||
|
|
||||||
|
3. **الحماية والاستمرارية (Data Persistence & High Availability):**
|
||||||
|
* عند تعطل الـ Pod المربوط بوحدة التخزين، تقوم منصة غنيمة تلقائياً بفصل قرص الـ Block Storage وإعادة ربطه (`Attach/Detach`) بالـ Pod الجديد عبر عقدة أخرى بدون أي فقدان للبيانات.
|
||||||
|
|
||||||
|
4. **النسخ الاحتياطي واللقطات الفورية (Snapshots & Replication):**
|
||||||
|
* يدعم Ghanimah Block Storage إنشاء لقطات فورية (Volume Snapshots) دورية بدون التأثير على أداء الخدمة الحية، مع إمكانية استرجاعها فوراً في حالات الطوارئ (Disaster Recovery).
|
||||||
337
q5-mithal-monitor/dashbourd.html
Normal file
@@ -0,0 +1,337 @@
|
|||||||
|
<!DOCTYPE html>
|
||||||
|
<html lang="en">
|
||||||
|
|
||||||
|
<head>
|
||||||
|
<meta charset="UTF-8">
|
||||||
|
<meta name="viewport" content="width=device-width, initial-scale=1.0">
|
||||||
|
|
||||||
|
<title>Mithal Monitoring Dashboard</title>
|
||||||
|
|
||||||
|
<script src="https://cdn.jsdelivr.net/npm/chart.js"></script>
|
||||||
|
|
||||||
|
<style>
|
||||||
|
* {
|
||||||
|
margin: 0;
|
||||||
|
padding: 0;
|
||||||
|
box-sizing: border-box;
|
||||||
|
font-family: Arial, Helvetica, sans-serif;
|
||||||
|
}
|
||||||
|
|
||||||
|
body {
|
||||||
|
background: #f4f6f8;
|
||||||
|
padding: 30px;
|
||||||
|
}
|
||||||
|
|
||||||
|
h1 {
|
||||||
|
margin-bottom: 25px;
|
||||||
|
}
|
||||||
|
|
||||||
|
.cards {
|
||||||
|
|
||||||
|
display: grid;
|
||||||
|
grid-template-columns: repeat(auto-fit, minmax(220px, 1fr));
|
||||||
|
gap: 20px;
|
||||||
|
margin-bottom: 30px;
|
||||||
|
|
||||||
|
}
|
||||||
|
|
||||||
|
.card {
|
||||||
|
|
||||||
|
background: white;
|
||||||
|
border-radius: 12px;
|
||||||
|
padding: 20px;
|
||||||
|
box-shadow: 0 3px 12px rgba(0, 0, 0, .1);
|
||||||
|
|
||||||
|
}
|
||||||
|
|
||||||
|
.card h3 {
|
||||||
|
|
||||||
|
margin-bottom: 10px;
|
||||||
|
color: #666;
|
||||||
|
|
||||||
|
}
|
||||||
|
|
||||||
|
.value {
|
||||||
|
|
||||||
|
font-size: 32px;
|
||||||
|
font-weight: bold;
|
||||||
|
|
||||||
|
}
|
||||||
|
|
||||||
|
.chart {
|
||||||
|
|
||||||
|
background: white;
|
||||||
|
padding: 20px;
|
||||||
|
border-radius: 12px;
|
||||||
|
box-shadow: 0 3px 12px rgba(0, 0, 0, .1);
|
||||||
|
margin-bottom: 30px;
|
||||||
|
|
||||||
|
}
|
||||||
|
|
||||||
|
table {
|
||||||
|
|
||||||
|
width: 100%;
|
||||||
|
border-collapse: collapse;
|
||||||
|
background: white;
|
||||||
|
|
||||||
|
}
|
||||||
|
|
||||||
|
th {
|
||||||
|
|
||||||
|
background: #202124;
|
||||||
|
color: white;
|
||||||
|
|
||||||
|
}
|
||||||
|
|
||||||
|
th,
|
||||||
|
td {
|
||||||
|
|
||||||
|
padding: 12px;
|
||||||
|
text-align: center;
|
||||||
|
border-bottom: 1px solid #ddd;
|
||||||
|
|
||||||
|
}
|
||||||
|
|
||||||
|
.ok {
|
||||||
|
|
||||||
|
color: green;
|
||||||
|
font-weight: bold;
|
||||||
|
|
||||||
|
}
|
||||||
|
|
||||||
|
.fail {
|
||||||
|
|
||||||
|
color: red;
|
||||||
|
font-weight: bold;
|
||||||
|
|
||||||
|
}
|
||||||
|
|
||||||
|
footer {
|
||||||
|
|
||||||
|
margin-top: 30px;
|
||||||
|
text-align: center;
|
||||||
|
color: #777;
|
||||||
|
|
||||||
|
}
|
||||||
|
</style>
|
||||||
|
|
||||||
|
</head>
|
||||||
|
|
||||||
|
<body>
|
||||||
|
|
||||||
|
<h1>📈 Mithal Monitoring Dashboard</h1>
|
||||||
|
|
||||||
|
<div class="cards">
|
||||||
|
|
||||||
|
<div class="card">
|
||||||
|
|
||||||
|
<h3>Uptime (24h)</h3>
|
||||||
|
|
||||||
|
<div class="value" id="uptime">--</div>
|
||||||
|
|
||||||
|
</div>
|
||||||
|
|
||||||
|
<div class="card">
|
||||||
|
|
||||||
|
<h3>Average Latency</h3>
|
||||||
|
|
||||||
|
<div class="value" id="latency">--</div>
|
||||||
|
|
||||||
|
</div>
|
||||||
|
|
||||||
|
<div class="card">
|
||||||
|
|
||||||
|
<h3>SSL Days Left</h3>
|
||||||
|
|
||||||
|
<div class="value" id="ssl">--</div>
|
||||||
|
|
||||||
|
</div>
|
||||||
|
|
||||||
|
<div class="card">
|
||||||
|
|
||||||
|
<h3>Total Checks</h3>
|
||||||
|
|
||||||
|
<div class="value" id="checks">--</div>
|
||||||
|
|
||||||
|
</div>
|
||||||
|
|
||||||
|
</div>
|
||||||
|
|
||||||
|
<div class="chart">
|
||||||
|
|
||||||
|
<canvas id="latencyChart"></canvas>
|
||||||
|
|
||||||
|
</div>
|
||||||
|
|
||||||
|
<h2 style="margin-bottom:15px;">Last 10 Checks</h2>
|
||||||
|
|
||||||
|
<table>
|
||||||
|
|
||||||
|
<thead>
|
||||||
|
|
||||||
|
<tr>
|
||||||
|
|
||||||
|
<th>Time</th>
|
||||||
|
<th>Status</th>
|
||||||
|
<th>Latency</th>
|
||||||
|
<th>DNS</th>
|
||||||
|
<th>SSL</th>
|
||||||
|
<th>Search</th>
|
||||||
|
|
||||||
|
</tr>
|
||||||
|
|
||||||
|
</thead>
|
||||||
|
|
||||||
|
<tbody id="tableBody">
|
||||||
|
|
||||||
|
</tbody>
|
||||||
|
|
||||||
|
</table>
|
||||||
|
|
||||||
|
<footer>
|
||||||
|
|
||||||
|
Refreshes every 60 seconds
|
||||||
|
|
||||||
|
</footer>
|
||||||
|
|
||||||
|
<script>
|
||||||
|
|
||||||
|
let chart;
|
||||||
|
|
||||||
|
async function loadMetrics() {
|
||||||
|
|
||||||
|
const response = await fetch("metrics.json");
|
||||||
|
|
||||||
|
const data = await response.json();
|
||||||
|
|
||||||
|
return data;
|
||||||
|
|
||||||
|
}
|
||||||
|
|
||||||
|
async function render() {
|
||||||
|
|
||||||
|
const data = await loadMetrics();
|
||||||
|
|
||||||
|
if (data.length === 0)
|
||||||
|
return;
|
||||||
|
|
||||||
|
const checks = data.length;
|
||||||
|
|
||||||
|
const uptime = data.filter(x => x.uptime).length;
|
||||||
|
|
||||||
|
const uptimePercent = ((uptime / checks) * 100).toFixed(2);
|
||||||
|
|
||||||
|
document.getElementById("uptime").innerHTML = uptimePercent + "%";
|
||||||
|
|
||||||
|
document.getElementById("checks").innerHTML = checks;
|
||||||
|
|
||||||
|
const avgLatency = (
|
||||||
|
|
||||||
|
data.reduce((a, b) => a + b.latency_ms, 0) / checks
|
||||||
|
|
||||||
|
).toFixed(2);
|
||||||
|
|
||||||
|
document.getElementById("latency").innerHTML = avgLatency + " ms";
|
||||||
|
|
||||||
|
document.getElementById("ssl").innerHTML =
|
||||||
|
|
||||||
|
data[data.length - 1].ssl.days_left + " days";
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
const lastHour = data.slice(-60);
|
||||||
|
|
||||||
|
const labels = lastHour.map(x => {
|
||||||
|
|
||||||
|
const d = new Date(x.timestamp);
|
||||||
|
|
||||||
|
return d.toLocaleTimeString();
|
||||||
|
|
||||||
|
});
|
||||||
|
|
||||||
|
const latency = lastHour.map(x => x.latency_ms);
|
||||||
|
|
||||||
|
if (chart)
|
||||||
|
chart.destroy();
|
||||||
|
|
||||||
|
chart = new Chart(
|
||||||
|
|
||||||
|
document.getElementById("latencyChart"),
|
||||||
|
|
||||||
|
{
|
||||||
|
|
||||||
|
type: "line",
|
||||||
|
|
||||||
|
data: {
|
||||||
|
|
||||||
|
labels: labels,
|
||||||
|
|
||||||
|
datasets: [{
|
||||||
|
|
||||||
|
label: "Latency (ms)",
|
||||||
|
|
||||||
|
data: latency,
|
||||||
|
|
||||||
|
fill: false,
|
||||||
|
|
||||||
|
tension: .2
|
||||||
|
|
||||||
|
}]
|
||||||
|
|
||||||
|
},
|
||||||
|
|
||||||
|
options: {
|
||||||
|
|
||||||
|
responsive: true
|
||||||
|
|
||||||
|
}
|
||||||
|
|
||||||
|
}
|
||||||
|
|
||||||
|
);
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
const tbody = document.getElementById("tableBody");
|
||||||
|
|
||||||
|
tbody.innerHTML = "";
|
||||||
|
|
||||||
|
data.slice(-10).reverse().forEach(item => {
|
||||||
|
|
||||||
|
tbody.innerHTML += `
|
||||||
|
|
||||||
|
<tr>
|
||||||
|
|
||||||
|
<td>${new Date(item.timestamp).toLocaleString()}</td>
|
||||||
|
|
||||||
|
<td class="${item.uptime ? 'ok' : 'fail'}">
|
||||||
|
|
||||||
|
${item.status_code}
|
||||||
|
|
||||||
|
</td>
|
||||||
|
|
||||||
|
<td>${item.latency_ms} ms</td>
|
||||||
|
|
||||||
|
<td>${item.dns_lookup_ms} ms</td>
|
||||||
|
|
||||||
|
<td>${item.ssl.days_left} days</td>
|
||||||
|
|
||||||
|
<td>${item.search.response_ms} ms</td>
|
||||||
|
|
||||||
|
</tr>
|
||||||
|
|
||||||
|
`;
|
||||||
|
|
||||||
|
});
|
||||||
|
|
||||||
|
}
|
||||||
|
|
||||||
|
render();
|
||||||
|
|
||||||
|
setInterval(render, 60000);
|
||||||
|
|
||||||
|
</script>
|
||||||
|
|
||||||
|
</body>
|
||||||
|
|
||||||
|
</html>
|
||||||
107
q5-mithal-monitor/metrics.json
Normal file
@@ -0,0 +1,107 @@
|
|||||||
|
[
|
||||||
|
{
|
||||||
|
"timestamp": "2026-07-28T09:59:01.075371",
|
||||||
|
"latency_ms": 1021.86,
|
||||||
|
"status_code": 200,
|
||||||
|
"uptime": true,
|
||||||
|
"dns_lookup_ms": 6.92,
|
||||||
|
"ssl": {
|
||||||
|
"expires_at": "2026-09-15T13:10:47",
|
||||||
|
"days_left": 49
|
||||||
|
},
|
||||||
|
"search": {
|
||||||
|
"status": 200,
|
||||||
|
"response_ms": 958.41
|
||||||
|
}
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"timestamp": "2026-07-28T10:00:03.486672",
|
||||||
|
"latency_ms": 1078.94,
|
||||||
|
"status_code": 200,
|
||||||
|
"uptime": true,
|
||||||
|
"dns_lookup_ms": 7.81,
|
||||||
|
"ssl": {
|
||||||
|
"expires_at": "2026-09-15T13:10:47",
|
||||||
|
"days_left": 49
|
||||||
|
},
|
||||||
|
"search": {
|
||||||
|
"status": 200,
|
||||||
|
"response_ms": 875.35
|
||||||
|
}
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"timestamp": "2026-07-28T10:01:05.991518",
|
||||||
|
"latency_ms": 1063.59,
|
||||||
|
"status_code": 200,
|
||||||
|
"uptime": true,
|
||||||
|
"dns_lookup_ms": 6.1,
|
||||||
|
"ssl": {
|
||||||
|
"expires_at": "2026-09-15T13:10:47",
|
||||||
|
"days_left": 49
|
||||||
|
},
|
||||||
|
"search": {
|
||||||
|
"status": 200,
|
||||||
|
"response_ms": 872.83
|
||||||
|
}
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"timestamp": "2026-07-28T10:02:08.495565",
|
||||||
|
"latency_ms": 1022.08,
|
||||||
|
"status_code": 200,
|
||||||
|
"uptime": true,
|
||||||
|
"dns_lookup_ms": 5.8,
|
||||||
|
"ssl": {
|
||||||
|
"expires_at": "2026-09-15T13:10:47",
|
||||||
|
"days_left": 49
|
||||||
|
},
|
||||||
|
"search": {
|
||||||
|
"status": 200,
|
||||||
|
"response_ms": 919.46
|
||||||
|
}
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"timestamp": "2026-07-28T10:03:10.815308",
|
||||||
|
"latency_ms": 944.67,
|
||||||
|
"status_code": 200,
|
||||||
|
"uptime": true,
|
||||||
|
"dns_lookup_ms": 7.7,
|
||||||
|
"ssl": {
|
||||||
|
"expires_at": "2026-09-15T13:10:47",
|
||||||
|
"days_left": 49
|
||||||
|
},
|
||||||
|
"search": {
|
||||||
|
"status": 200,
|
||||||
|
"response_ms": 883.18
|
||||||
|
}
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"timestamp": "2026-07-28T10:04:12.942249",
|
||||||
|
"latency_ms": 809.59,
|
||||||
|
"status_code": 200,
|
||||||
|
"uptime": true,
|
||||||
|
"dns_lookup_ms": 5.28,
|
||||||
|
"ssl": {
|
||||||
|
"expires_at": "2026-09-15T13:10:47",
|
||||||
|
"days_left": 49
|
||||||
|
},
|
||||||
|
"search": {
|
||||||
|
"status": 200,
|
||||||
|
"response_ms": 849.83
|
||||||
|
}
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"timestamp": "2026-07-28T10:05:15.376909",
|
||||||
|
"latency_ms": 909.74,
|
||||||
|
"status_code": 200,
|
||||||
|
"uptime": true,
|
||||||
|
"dns_lookup_ms": 4.95,
|
||||||
|
"ssl": {
|
||||||
|
"expires_at": "2026-09-15T13:10:47",
|
||||||
|
"days_left": 49
|
||||||
|
},
|
||||||
|
"search": {
|
||||||
|
"status": 200,
|
||||||
|
"response_ms": 803.56
|
||||||
|
}
|
||||||
|
}
|
||||||
|
]
|
||||||
204
q5-mithal-monitor/monitor.py
Normal file
@@ -0,0 +1,204 @@
|
|||||||
|
import requests
|
||||||
|
import socket
|
||||||
|
import ssl
|
||||||
|
import json
|
||||||
|
import os
|
||||||
|
import time
|
||||||
|
from datetime import datetime
|
||||||
|
|
||||||
|
# -----------------------------
|
||||||
|
# Configuration
|
||||||
|
# -----------------------------
|
||||||
|
BASE_URL = "https://mithal.space"
|
||||||
|
SEARCH_URL = "https://mithal.space/search?q=test"
|
||||||
|
|
||||||
|
OUTPUT_FILE = "metrics.json"
|
||||||
|
INTERVAL = 60 # seconds
|
||||||
|
|
||||||
|
MAX_RECORDS = 1440 # 24 hours (1 record/min)
|
||||||
|
|
||||||
|
|
||||||
|
# -----------------------------
|
||||||
|
# HTTP Latency + Uptime
|
||||||
|
# -----------------------------
|
||||||
|
def check_http():
|
||||||
|
start = time.perf_counter()
|
||||||
|
|
||||||
|
try:
|
||||||
|
response = requests.get(BASE_URL, timeout=10)
|
||||||
|
|
||||||
|
latency = round((time.perf_counter() - start) * 1000, 2)
|
||||||
|
|
||||||
|
return {
|
||||||
|
"status_code": response.status_code,
|
||||||
|
"uptime": response.status_code == 200,
|
||||||
|
"latency_ms": latency
|
||||||
|
}
|
||||||
|
|
||||||
|
except Exception:
|
||||||
|
latency = round((time.perf_counter() - start) * 1000, 2)
|
||||||
|
|
||||||
|
return {
|
||||||
|
"status_code": None,
|
||||||
|
"uptime": False,
|
||||||
|
"latency_ms": latency
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
# -----------------------------
|
||||||
|
# DNS Lookup Time
|
||||||
|
# -----------------------------
|
||||||
|
def check_dns():
|
||||||
|
host = BASE_URL.replace("https://", "").replace("http://", "").split("/")[0]
|
||||||
|
|
||||||
|
start = time.perf_counter()
|
||||||
|
|
||||||
|
try:
|
||||||
|
socket.gethostbyname(host)
|
||||||
|
dns_time = round((time.perf_counter() - start) * 1000, 2)
|
||||||
|
|
||||||
|
except Exception:
|
||||||
|
dns_time = None
|
||||||
|
|
||||||
|
return dns_time
|
||||||
|
|
||||||
|
|
||||||
|
# -----------------------------
|
||||||
|
# SSL Expiry
|
||||||
|
# -----------------------------
|
||||||
|
def check_ssl():
|
||||||
|
host = BASE_URL.replace("https://", "").replace("http://", "").split("/")[0]
|
||||||
|
|
||||||
|
try:
|
||||||
|
context = ssl.create_default_context()
|
||||||
|
|
||||||
|
with context.wrap_socket(
|
||||||
|
socket.socket(socket.AF_INET),
|
||||||
|
server_hostname=host
|
||||||
|
) as s:
|
||||||
|
|
||||||
|
s.settimeout(10)
|
||||||
|
s.connect((host, 443))
|
||||||
|
|
||||||
|
cert = s.getpeercert()
|
||||||
|
|
||||||
|
expire = datetime.strptime(
|
||||||
|
cert["notAfter"],
|
||||||
|
"%b %d %H:%M:%S %Y %Z"
|
||||||
|
)
|
||||||
|
|
||||||
|
remaining = (expire - datetime.utcnow()).days
|
||||||
|
|
||||||
|
return {
|
||||||
|
"expires_at": expire.isoformat(),
|
||||||
|
"days_left": remaining
|
||||||
|
}
|
||||||
|
|
||||||
|
except Exception:
|
||||||
|
|
||||||
|
return {
|
||||||
|
"expires_at": None,
|
||||||
|
"days_left": None
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
# -----------------------------
|
||||||
|
# Search Response Time
|
||||||
|
# -----------------------------
|
||||||
|
def check_search():
|
||||||
|
start = time.perf_counter()
|
||||||
|
|
||||||
|
try:
|
||||||
|
response = requests.get(
|
||||||
|
SEARCH_URL,
|
||||||
|
timeout=10
|
||||||
|
)
|
||||||
|
|
||||||
|
elapsed = round((time.perf_counter() - start) * 1000, 2)
|
||||||
|
|
||||||
|
return {
|
||||||
|
"status": response.status_code,
|
||||||
|
"response_ms": elapsed
|
||||||
|
}
|
||||||
|
|
||||||
|
except Exception:
|
||||||
|
|
||||||
|
elapsed = round((time.perf_counter() - start) * 1000, 2)
|
||||||
|
|
||||||
|
return {
|
||||||
|
"status": None,
|
||||||
|
"response_ms": elapsed
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
# -----------------------------
|
||||||
|
# Load JSON
|
||||||
|
# -----------------------------
|
||||||
|
def load_metrics():
|
||||||
|
|
||||||
|
if not os.path.exists(OUTPUT_FILE):
|
||||||
|
return []
|
||||||
|
|
||||||
|
try:
|
||||||
|
with open(OUTPUT_FILE, "r") as f:
|
||||||
|
return json.load(f)
|
||||||
|
|
||||||
|
except Exception:
|
||||||
|
return []
|
||||||
|
|
||||||
|
|
||||||
|
# -----------------------------
|
||||||
|
# Save JSON
|
||||||
|
# -----------------------------
|
||||||
|
def save_metrics(data):
|
||||||
|
|
||||||
|
with open(OUTPUT_FILE, "w") as f:
|
||||||
|
json.dump(data, f, indent=4)
|
||||||
|
|
||||||
|
|
||||||
|
# -----------------------------
|
||||||
|
# Main Monitor Loop
|
||||||
|
# -----------------------------
|
||||||
|
def monitor():
|
||||||
|
|
||||||
|
print("Starting Mithal Monitor...")
|
||||||
|
|
||||||
|
while True:
|
||||||
|
|
||||||
|
http = check_http()
|
||||||
|
dns = check_dns()
|
||||||
|
ssl_info = check_ssl()
|
||||||
|
search = check_search()
|
||||||
|
|
||||||
|
record = {
|
||||||
|
"timestamp": datetime.utcnow().isoformat(),
|
||||||
|
|
||||||
|
"latency_ms": http["latency_ms"],
|
||||||
|
|
||||||
|
"status_code": http["status_code"],
|
||||||
|
|
||||||
|
"uptime": http["uptime"],
|
||||||
|
|
||||||
|
"dns_lookup_ms": dns,
|
||||||
|
|
||||||
|
"ssl": ssl_info,
|
||||||
|
|
||||||
|
"search": search
|
||||||
|
}
|
||||||
|
|
||||||
|
data = load_metrics()
|
||||||
|
|
||||||
|
data.append(record)
|
||||||
|
|
||||||
|
if len(data) > MAX_RECORDS:
|
||||||
|
data = data[-MAX_RECORDS:]
|
||||||
|
|
||||||
|
save_metrics(data)
|
||||||
|
|
||||||
|
print(record)
|
||||||
|
|
||||||
|
time.sleep(INTERVAL)
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
monitor()
|
||||||