diff --git a/README.md b/README.md index 73f466b..503f212 100644 --- a/README.md +++ b/README.md @@ -1,28 +1,187 @@ -# مشروع تدريب: نشر ومراقبة على Ghaymah — الحل الكامل +# Ghaymah SRE Technical Assessment -هذا المشروع يغطي الأنشطة الخمسة المطلوبة: +This repository contains my submission for the **Ghaymah Internship Program – SRE Technical Assessment**. -| # | المجلد | المهمة | -|---|---|---| -| 1 | `task1-deploy/` | Dockerfile + API + /health + سكريبت مراقبة + dashboard | -| 2 | `task2-postmortem/` | تقرير Postmortem لحادثة OOMKilled + سياسة auto-scaling | -| 3 | `task3-cicd/` | GitHub Actions workflow + شرح staging/production + ghaymah CLI | -| 4 | `task4-scalability/` | Architecture diagram + حسابات 15,000 req/s + Cold start + Block Storage | -| 5 | `task5-mithal-dashboard/` | جامع مقاييس + dashboard لموقع mithal.space | +The project demonstrates deployment, monitoring, incident analysis, CI/CD automation, scalability planning, and monitoring dashboard implementation using Ghaymah Cloud Platform. -## ⚠️ ملاحظة مهمة عن أوامر Ghaymah CLI -لم أستطع الوصول لتوثيق تفصيلي حي لأوامر `ghaymah` CLI الدقيقة (الموقع الرسمي -docs.ghaymah.cloud مبني بجافاسكربت ولا يظهر المحتوى الكامل عبر البحث). لذلك -كل أوامر `ghaymah ...` في هذا المشروع هي **بالنمط القياسي المتوقع** لمنصات -الحاويات (login → build/push → deploy → status/logs → rollback)، وتحتاج منك -التأكد من الصياغة الدقيقة من https://docs.ghaymah.cloud أو من CLI نفسه عبر -`ghaymah --help` قبل الاستخدام الفعلي في التسليم. +--- -## ترتيب مقترح للعمل -1. جرّب `task1-deploy` محلياً بـ Docker أولاً، ثم انشرها فعلياً على Ghaymah. -2. شغّل `monitor.sh` على الرابط المنشور وافتح `dashboard.html`. -3. اقرأ `task2-postmortem/postmortem.md` وطبّق سياسة الـ auto-scaling على نفس التطبيق إن أمكن. -4. أعدّ الـ CI/CD (`task3-cicd`) وفعّل الموافقة اليدوية في GitHub Environments. -5. استخدم حسابات `task4-scalability/scalability.md` كجزء من التقرير الكتابي. -6. شغّل `task5-mithal-dashboard/collector.py` وانشر اللوحة كما في المهمة 1. -# Ghaymah-sre-tasks +# Repository Structure + +``` +. +├── q1-deploy-monitor/ +│ ├── Dockerfile +│ ├── health-check.sh +│ ├── dashboard.html +│ └── README.md +│ +├── q2-postmortem/ +│ ├── postmortem-report.md +│ └── README.md +│ +├── q3-cicd/ +│ ├── workflow.yml +│ └── README.md +│ +├── q4-scalability/ +│ ├── architecture.png +│ ├── calculations.md +│ └── README.md +│ +├── q5-mithal-monitor/ +│ ├── collector.py +│ ├── dashboard.html +│ ├── Dockerfile +│ ├── requirements.txt +│ └── README.md +│ +├── common-mortakaz/ +├── common-qabilah/ +└── README.md +``` + +--- + +# Task 1 — Deploy & Monitor + +### Objective + +Deploy a containerized web application on **Ghaymah Cloud** and implement a basic monitoring dashboard. + +### Deliverables + +- Dockerized application +- Health check endpoint +- Monitoring dashboard +- Successful deployment on Ghaymah + +--- + +# Task 2 — Incident Postmortem + +### Objective + +Analyze a production outage caused by repeated **OOMKilled** events. + +### Deliverables + +- Executive summary +- Timeline +- Root cause analysis +- Corrective actions +- Preventive recommendations +- Auto-scaling proposal + +--- + +# Task 3 — CI/CD Pipeline + +### Objective + +Create a complete deployment pipeline using **GitHub Actions** and **Ghaymah CLI**. + +### Features + +- Docker image build +- Application validation +- Deploy to Staging +- Manual Approval +- Deploy to Production +- Health checks + +--- + +# Task 4 — Scalability Design + +### Objective + +Design infrastructure capable of serving **15,000 Requests/sec**. + +### Includes + +- Architecture diagram +- Capacity calculations +- Horizontal scaling strategy +- Cold-start optimization +- Block Storage usage +- Auto-scaling recommendations + +--- + +# Task 5 — Mithal Monitoring Dashboard + +### Objective + +Build a monitoring solution for **https://mithal.space** + +### Metrics Collected + +- HTTP Latency +- Uptime +- DNS Resolution Time +- SSL Certificate Status +- Search Response Time + +### Dashboard Features + +- 24-hour uptime percentage +- Latency chart +- SSL expiration status +- Latest monitoring checks +- Automatic refresh + +--- + +# Technologies Used + +- Python +- Docker +- GitHub Actions +- Ghaymah CLI +- HTML +- CSS +- JavaScript +- Chart.js + +--- + +# Deployment + +Applications were deployed using the official **Ghaymah CLI**. + +General deployment workflow: + +```bash +gy auth login + +gy resource app init --project-id + +gy resource app launch +``` + +--- + +# Notes + +- Each task is self-contained. +- Every task includes its own documentation. +- The implementation follows the official Ghaymah documentation whenever applicable. + +--- + +# Author + +**Moamen Lotfy** + +SRE Track Candidate + +Ghaymah Internship Program + +## Qabilah Profile + +As required by the technical assessment, I created a Qabilah account and followed the official Ghaymah account. + +**Profile Link:** + +https://qabilah.com/profile/momendevops/professional-profile diff --git a/common-mortakaz/integration-1.md b/common-mortakaz/integration-1.md new file mode 100644 index 0000000..1ce0359 --- /dev/null +++ b/common-mortakaz/integration-1.md @@ -0,0 +1,91 @@ +# Integration Proposal: Mando × Ghaymah + +## Product Overview + +Mando is an AI-powered customer support platform designed for small and medium-sized businesses. It automates customer service operations, reduces response time, lowers operational costs, and improves customer satisfaction. The platform also supports managing multiple organizations from a single dashboard, making it suitable for agencies and businesses serving multiple clients. + +--- + +# Proposed Integration with Ghaymah + +The proposed integration is to deploy and operate Mando on Ghaymah Cloud using its container-based application platform. + +The deployment workflow would use GitHub Actions together with Ghaymah CLI to automatically build, test, and deploy every new version of Mando. + +Ghaymah provides: + +- Containerized deployment +- Automated application delivery +- Scalable infrastructure +- Centralized application management +- Monitoring and logging capabilities +- Secure cloud hosting in the Arab region + +Deployment flow: + +Developer → GitHub → GitHub Actions → Ghaymah CLI → Ghaymah Platform + +--- + +# Value for End Users + +This integration provides several benefits: + +- Faster platform updates through CI/CD. +- Higher application availability. +- Reduced service interruptions. +- Better response time for users in the Middle East. +- Ability to scale automatically during periods of high customer traffic. +- Simplified infrastructure management, allowing the Mando team to focus on product development instead of server administration. + +--- + +# Architecture Sketch + +```text + +--------------------+ + | Developers | + +---------+----------+ + | + v + +--------------------+ + | GitHub | + +---------+----------+ + | + GitHub Actions CI/CD + | + v + +--------------------+ + | Ghaymah CLI | + +---------+----------+ + | + v + +--------------------+ + | Ghaymah Platform | + | Containers & Apps | + +---------+----------+ + | + v + +--------------------+ + | Mando Platform | + +---------+----------+ + | + v + End Users +``` + +--- + +# Potential Challenges + +## Technical + +- Secure authentication between GitHub Actions and Ghaymah CLI. +- Managing secrets and environment variables. +- Scaling AI workloads during traffic spikes. + +## Business + +- Infrastructure migration from existing cloud providers. +- Training the engineering team on Ghaymah deployment workflows. +- Planning a gradual migration to minimize downtime. diff --git a/common-mortakaz/integration-2.md b/common-mortakaz/integration-2.md new file mode 100644 index 0000000..f3c6427 --- /dev/null +++ b/common-mortakaz/integration-2.md @@ -0,0 +1,82 @@ +# Integration Proposal: Dovera.ai × Mithal Space + +## Product Overview + +Dovera.ai is an AI-powered platform that automates sales, marketing, and customer experience. It helps businesses generate leads, engage customers, automate communication, and improve marketing efficiency using artificial intelligence. + +--- + +# Proposed Integration with Mithal Space + +Instead of maintaining its own language models, Dovera.ai can integrate with Mithal Space APIs to provide advanced Arabic AI capabilities. + +The integration would allow Dovera.ai to use Mithal for: + +- AI-generated sales messages +- Arabic customer support +- Marketing content generation +- Conversation summarization +- Sentiment analysis +- Intelligent response suggestions + +This approach enables Dovera.ai to continuously benefit from improvements made to Mithal's Arabic language models without maintaining complex AI infrastructure internally. + +--- + +# Value for End Users + +Users would benefit from: + +- More accurate Arabic content generation. +- Faster AI responses. +- Improved customer engagement. +- Better quality marketing campaigns. +- Lower service costs. +- Continuous AI model improvements without service interruption. + +--- + +# Architecture Sketch + +```text + +------------------+ + | Customer | + +--------+---------+ + | + v + +------------------+ + | Dovera.ai | + +--------+---------+ + | + API Requests + | + v + +------------------+ + | Mithal Space | + | Arabic AI Models | + +--------+---------+ + | + v + AI Generated Output + | + v + End Customer +``` + +--- + +# Potential Challenges + +## Technical + +- API latency during peak traffic. +- Rate limiting. +- Handling sensitive customer information securely. +- Maintaining compatibility with future API versions. + +## Business + +- API usage costs. +- Vendor dependency. +- Service Level Agreements (SLA). +- Data residency and compliance requirements. diff --git a/common-qabilah/qabilah-profile.txt b/common-qabilah/qabilah-profile.txt new file mode 100644 index 0000000..9829a71 --- /dev/null +++ b/common-qabilah/qabilah-profile.txt @@ -0,0 +1,10 @@ +Qabilah Profile + +Name: Moamen Lotfy + +Profile: +https://qabilah.com/profile/momendevops/professional-profile + +Status: +✔ Created Qabilah account +✔ Followed the official Ghaymah account diff --git a/.github/workflows/deploy.yml b/task3-cicd/.github/workflows/deploy.yml similarity index 100% rename from .github/workflows/deploy.yml rename to task3-cicd/.github/workflows/deploy.yml diff --git a/task4-scalability/calculations.md b/task4-scalability/calculations.md new file mode 100644 index 0000000..b0dddd9 --- /dev/null +++ b/task4-scalability/calculations.md @@ -0,0 +1,259 @@ +# Task 4 — Scalability & Load Distribution (15,000 Requests/Second) + +## Overview + +This document proposes a scalable architecture for deploying an application on **Ghaymah Cloud** capable of handling **15,000 HTTP requests per second** while maintaining high availability, fault tolerance, and low response latency. + +--- + +# 1. High-Level Architecture + +```mermaid +flowchart TD + + Users[Users] + + Users --> DNS[Ghaymah DNS / Edge] + + DNS --> LB[Load Balancer] + + LB --> A1[Container 1] + LB --> A2[Container 2] + LB --> A3[Container 3] + LB --> A4[...] + LB --> A39[Container 39] + + A1 --> Redis[(Redis Cache)] + A2 --> Redis + A3 --> Redis + A39 --> Redis + + Redis --> DB[(Primary Database)] + + DB --> Storage[(Ghaymah Block Storage)] + + subgraph AutoScaling + Metrics[CPU • Memory • RPS] + Decision{Scale?} + ScaleOut[Add Containers] + ScaleIn[Remove Containers] + + Metrics --> Decision + Decision -->|High Load| ScaleOut + Decision -->|Low Load| ScaleIn + end + + LB -. Metrics .-> Metrics + ScaleOut -. Register New Instances .-> LB +``` + +--- + +# 2. Capacity Calculation + +### Given + +- Expected traffic = **15,000 requests/second** +- One container capacity = **500 requests/second** + +### Required Containers + +``` +15,000 / 500 = 30 Containers +``` + +To avoid saturation during sudden traffic spikes or node failures, a **30% safety margin** is added. + +``` +30 × 1.30 = 39 Containers +``` + +## Final Capacity + +| Item | Value | +|-------|------:| +| Target Load | 15,000 req/s | +| Capacity per Container | 500 req/s | +| Base Containers | 30 | +| Safety Margin | 30% | +| Recommended Maximum | **39 Containers** | + +--- + +# 3. Auto Scaling Policy + +Recommended configuration: + +```yaml +min_replicas: 12 +max_replicas: 39 + +target_cpu: 65% +target_memory: 70% + +target_requests_per_container: 400 + +scale_up: + increase: 4 containers + cooldown: 60s + +scale_down: + decrease: 2 containers + cooldown: 300s +``` + +### Scaling Rules + +Scale Out when: + +- CPU > 65% +- Memory > 70% +- Average Requests > 400 req/s per container + +Scale In when: + +- CPU < 35% +- Memory < 40% +- Traffic remains low for 5 minutes + +This policy minimizes unnecessary scaling operations while maintaining performance. + +--- + +# 4. Cold Start Strategy + +Launching new containers requires image download, application startup, and service initialization. + +To reduce startup latency: + +### 1. Readiness Probe + +New containers receive traffic **only after** passing repeated `/health` checks. + +Example: + +- Interval: 5 seconds +- Success Threshold: 3 + +--- + +### 2. Pre-Warmed Containers + +Keep a minimum of **12 running replicas** to absorb sudden traffic spikes without waiting for new containers to start. + +--- + +### 3. Lightweight Docker Images + +Use slim base images such as: + +```text +python:3.11-slim +``` + +Smaller images reduce image pull time significantly. + +--- + +### 4. Predictive Scaling + +Scale based on traffic trends rather than waiting until CPU reaches its maximum threshold. + +--- + +### 5. Connection Draining + +Before terminating a container: + +- Stop accepting new requests. +- Finish active requests. +- Remove the container gracefully. + +This prevents user-facing errors during scale-down. + +--- + +# 5. Load Balancing Strategy + +The Load Balancer should distribute requests evenly across all healthy containers. + +Recommended algorithm: + +- Round Robin +- Least Connections (preferred for variable workloads) + +Health checks should continuously monitor: + +- `/health` +- Container availability +- Response latency + +Unhealthy containers should automatically be removed from rotation. + +--- + +# 6. High Availability + +To improve reliability: + +- Deploy multiple container replicas. +- Eliminate single points of failure. +- Automatically replace unhealthy containers. +- Use horizontal scaling instead of vertical scaling. +- Keep the application stateless whenever possible. + +--- + +# 7. Using Ghaymah Block Storage + +The application layer remains **stateless**, allowing containers to be created or removed without affecting user requests. + +Persistent storage is required for: + +- User uploads +- Database storage +- Persistent logs +- Queue data +- Shared application files + +Ghaymah Block Storage provides durable volumes that remain available even if containers are recreated. + +Benefits include: + +- Persistent data +- Easy attachment to new containers +- Independent lifecycle from application containers +- Simplified disaster recovery + +--- + +# 8. Design Summary + +| Component | Purpose | +|-----------|---------| +| Ghaymah DNS | Entry point | +| Load Balancer | Traffic distribution | +| 39 Containers | Handle application traffic | +| Redis Cache | Reduce database load | +| Primary Database | Persistent data | +| Ghaymah Block Storage | Durable storage | +| Auto Scaling | Automatic scaling | +| Health Checks | Availability monitoring | + +--- + +# Conclusion + +The proposed architecture can reliably handle **15,000 requests per second** using horizontal scaling on Ghaymah Cloud. + +Key design principles include: + +- Horizontal scalability +- Automatic load balancing +- Redis caching +- Health checks +- Automatic scaling +- Graceful container lifecycle +- Persistent storage using Ghaymah Block Storage + +This architecture delivers high availability, fault tolerance, and consistent performance under heavy production workloads. diff --git a/task4-scalability/scalability.md b/task4-scalability/scalability.md deleted file mode 100644 index 3aa1381..0000000 --- a/task4-scalability/scalability.md +++ /dev/null @@ -1,113 +0,0 @@ -# المهمة 4 — قابلية التوسع وتوزيع الأحمال (15,000 req/s) - -## 1) Architecture Diagram - -```mermaid -flowchart TD - U[المستخدمون] --> DNS[Ghaymah DNS / CDN Edge] - DNS --> LB[Load Balancer
غيمة] - - LB --> C1[حاوية 1
500 req/s] - LB --> C2[حاوية 2
500 req/s] - LB --> C3[حاوية 3
500 req/s] - LB --> C4[...] - LB --> C39[حاوية 39
500 req/s] - - C1 --> CACHE[(Redis Cache)] - C2 --> CACHE - C3 --> CACHE - C39 --> CACHE - - CACHE --> DB[(قاعدة بيانات رئيسية)] - C1 --> BLOCK[(Ghaymah Block Storage
للبيانات الدائمة)] - C39 --> BLOCK - - subgraph Autoscaler["Auto-scaling Controller"] - METRICS[مقاييس CPU/Memory/RPS] --> DECIDE{تجاوز الحد؟} - DECIDE -- نعم --> ADD[إضافة حاويات جديدة] - DECIDE -- لا --> KEEP[إبقاء العدد الحالي] - end - - LB -. تقرير المقاييس .-> METRICS - ADD -. توسيع .-> LB -``` - -**الفكرة:** طلبات المستخدمين تصل أولاً لـ Load Balancer الذي يوزّعها على مجموعة -حاويات متطابقة (horizontal scaling)، كل حاوية تتعامل مع الحمل الخاص بها وتتصل -بطبقة cache مشتركة قبل قاعدة البيانات لتقليل الضغط عليها، بينما البيانات الدائمة -(uploads, session files, ...) تُخزَّن على Ghaymah Block Storage القابل للربط -بأي حاوية جديدة. - ---- - -## 2) حساب عدد الحاويات المطلوبة - -المعطيات: -- الحمل المستهدف: **15,000 req/s** -- سعة الحاوية الواحدة: **500 req/s** -- هامش أمان: **30%** (لتفادي التشبع عند تذبذب الحمل أو فقدان حاوية) - -**الحساب:** -``` -عدد الحاويات الأساسي = 15,000 / 500 = 30 حاوية - -مع هامش الأمان 30%: -30 × 1.30 = 39 حاوية -``` - -➡️ **العدد المطلوب فعلياً = 39 حاوية** (وليس 30)، بحيث لو سقطت بضع حاويات -أو ارتفع الحمل مؤقتاً 20-30% فوق المتوقع، النظام لا يصل للتشبع الكامل. - -**توصية عملية للإعداد:** -```yaml -min_replicas: 12 # يغطي حمل القاعدة العادي (ليس ذروة اليوم) -max_replicas: 39 # يغطي ذروة 15,000 req/s + الهامش -target_cpu: 65% -target_rps_per_pod: 400 # هدف أقل من السعة القصوى (500) لإعطاء هامش استجابة -``` - ---- - -## 3) استراتيجية Cold Start للحاويات الجديدة - -المشكلة: حاوية جديدة تحتاج وقتاً (تحميل صورة، تهيئة التطبيق، اتصال بقاعدة -البيانات) قبل أن تكون جاهزة فعلياً لاستقبال حمل — لو أرسل لها الـ LB طلبات -فوراً ستفشل أو تبطئ الاستجابة. - -الحل المقترح: -1. **Readiness Probe صارم:** لا يُضاف الـ pod لقائمة الـ Load Balancer إلا - بعد نجاح فحص `/health` عدة مرات متتالية (مثلاً 3 نجاحات متتالية كل 5 ثوانٍ). -2. **Pre-warmed pool (نسخ دافئة جاهزة):** الاحتفاظ بعدد أدنى من الحاويات - (`min_replicas`) يعمل باستمرار حتى في أوقات الحمل المنخفض، بدل الاعتماد - بالكامل على scale-from-zero، لأن بدء التشغيل من الصفر أبطأ بكثير. -2. **صور خفيفة (slim images):** استخدام صور أساس صغيرة (مثل `python:3.11-slim`) - لتقليل وقت سحب الصورة (image pull time) عند جدولة حاوية جديدة على node جديد. -3. **Predictive/Proactive scaling:** التوسع بناءً على اتجاه الحمل (trend) - وليس فقط عندما يتجاوز الحمل الحد الحالي — مثال: لو الحمل يرتفع بمعدل ثابت - خلال آخر دقيقتين، ابدأ بإضافة حاويات الآن بدل الانتظار حتى الوصول للحد الأقصى. -4. **Connection draining عند الإزالة:** عند تقليص العدد (scale-down)، إعطاء - الحاوية مهلة لإنهاء الطلبات الجارية قبل إيقافها فعلياً، لتفادي أخطاء للمستخدمين. - ---- - -## 4) استخدام Ghaymah Block Storage لأحمال العمل ذات الحالة (Stateful) - -التطبيق نفسه (API) عادة **stateless** — أي حاوية جديدة يمكن أن تخدم أي طلب -دون الحاجة لبيانات محفوظة محلياً. لكن بعض المكونات تحتاج تخزيناً دائماً: - -- **ملفات مرفوعة من المستخدمين** (صور، مستندات) يجب أن تبقى متاحة حتى لو - تغيّرت الحاوية التي تخدم الطلب التالي. -- **قواعد بيانات أو أنظمة queue** تحتاج تخزيناً لا يُفقد عند إعادة تشغيل الحاوية. -- **ملفات cache دائمة أو logs** يُراد الاحتفاظ بها عبر إعادة الجدولة. - -**كيف يُستخدم Ghaymah Block Storage هنا:** -- يُربط (mount) كـ volume دائم لأي حاوية تحتاج تخزيناً (مثل خدمة قاعدة - البيانات أو خدمة رفع الملفات)، بحيث تبقى البيانات موجودة حتى لو حُذفت - الحاوية وأُعيد إنشاؤها من جديد على node مختلف. -- يُفصل عن الحاويات نفسها (decoupled)، فيمكن لأي نسخة جديدة من الخدمة أن - "ترث" نفس البيانات بمجرد إعادة ربط نفس الـ volume، بدل تخزين البيانات - داخل الحاوية (وهو ما يفقد عند إعادة التشغيل). -- بالنسبة للـ API نفسه (الطبقة الأمامية عالية التوسع من 39 حاوية)، يبقى - **stateless تماماً** ولا يحتاج Block Storage مباشرة — فقط الطبقات الخلفية - (قاعدة البيانات، تخزين الملفات) هي من تحتاجه، مما يسمح للطبقة الأمامية - بالتوسع والتقلص بحرية دون القلق على فقدان بيانات. diff --git a/task5-mithal-dashboard/.ghaymah.json b/task5-mithal-dashboard/.ghaymah.json new file mode 100644 index 0000000..403985f --- /dev/null +++ b/task5-mithal-dashboard/.ghaymah.json @@ -0,0 +1,17 @@ +{ + "id": "e5a04904-833b-4348-9690-8c26ffb3caef", + "name": "task5-mithal-dashboard", + "projectId": "34340b16-24f2-4db9-89e4-bf94bb372e68", + "ports": [ + { + "expose": true, + "number": 80 + } + ], + "publicAccess": { + "enabled": true, + "domain": "auto" + }, + "resourceTier": "t1", + "dockerFileName": "Dockerfile" +} \ No newline at end of file diff --git a/task5-mithal-dashboard/Dockerfile b/task5-mithal-dashboard/Dockerfile new file mode 100644 index 0000000..2ef49f1 --- /dev/null +++ b/task5-mithal-dashboard/Dockerfile @@ -0,0 +1,11 @@ +FROM nginx:alpine + +WORKDIR /usr/share/nginx/html + +COPY dashboard.html . +COPY latest.json . +COPY history.json . + +EXPOSE 80 + +CMD ["nginx", "-g", "daemon off;"] diff --git a/task5-mithal-dashboard/README.md b/task5-mithal-dashboard/README.md index 140ac17..6eff282 100644 --- a/task5-mithal-dashboard/README.md +++ b/task5-mithal-dashboard/README.md @@ -1,62 +1,223 @@ -# المهمة 5 — لوحة مراقبة mithal.space +# Task 5 – Monitoring Dashboard for mithal.space -## الملفات -- `collector.py` — يجمع المقاييس كل دقيقة (latency, uptime, SSL, DNS, search) ويخزنها في: - - `metrics.csv` (سجل تاريخي كامل، صف لكل فحص) - - `latest.json` (أحدث قراءة، يقرأها الـ dashboard) - - `history.json` (آخر 1440 قراءة = 24 ساعة، يقرأها الـ dashboard للرسم البياني والجدول) -- `dashboard.html` — يعرض uptime%، رسم بياني لزمن الاستجابة، حالة SSL، وسجل آخر 10 فحوصات -- `requirements.txt` — اعتماديات بايثون (`requests`) +## Overview + +This project implements a lightweight monitoring solution for **mithal.space**. It periodically collects website health metrics, stores them in CSV and JSON formats, and displays the results in a real-time monitoring dashboard. + +The solution was built as part of the **Ghaymah Cloud Training Project – Task 5**. + +--- + +# Features + +The monitoring system collects the following metrics every **60 seconds**: + +* HTTP Response Latency +* Website Availability (Uptime) +* HTTP Status Code +* SSL Certificate Status +* SSL Certificate Expiration +* DNS Resolution Time +* Search Endpoint Response Time + +All collected metrics are automatically stored for visualization and historical analysis. + +--- + +# Project Structure + +```text +task5-mithal-dashboard/ +│ +├── collector.py # Monitoring data collector +├── dashboard.html # Monitoring dashboard +├── metrics.csv # Historical CSV data +├── latest.json # Latest collected metrics +├── history.json # Historical JSON records +├── requirements.txt +├── Dockerfile +└── README.md +``` + +--- + +# Collected Metrics + +| Metric | Description | +| --------------- | ----------------------------------------- | +| Latency | HTTP response time in milliseconds | +| Uptime | Website availability based on HTTP status | +| Status Code | HTTP response code | +| DNS | DNS lookup time | +| SSL Status | Certificate validity | +| SSL Expiration | Remaining days until expiration | +| Search Response | Search endpoint response time | + +--- + +# Data Storage + +The collector stores monitoring data in three formats: + +### metrics.csv + +Historical metrics suitable for spreadsheet analysis. + +### latest.json + +Contains only the latest monitoring result. + +### history.json + +Stores the monitoring history for the last 24 hours (1440 checks). + +--- + +# Dashboard Features + +The dashboard provides: + +* 24-hour Uptime Percentage +* Current Website Status +* SSL Certificate Status +* Remaining SSL Validity +* DNS Response Time +* Average HTTP Latency +* Last Check Timestamp +* Response Time Line Chart (Last Hour) +* Latest 10 Monitoring Results +* Auto Refresh every 60 seconds + +--- + +# Technologies + +* Python 3 +* HTML5 +* CSS3 +* JavaScript +* Chart.js +* Requests Library + +--- + +# Installation + +Install the required Python packages: -## 1) التشغيل محلياً ```bash -cd task5-mithal-dashboard pip install -r requirements.txt +``` -# فحص واحد للتجربة +--- + +# Run One Check + +```bash python collector.py --once +``` -# تشغيل مستمر (يفحص كل دقيقة) -python collector.py & +--- -# تقديم الملفات (dashboard.html + json files) عبر خادم بسيط +# Continuous Monitoring + +```bash +python collector.py +``` + +The collector performs a new health check every 60 seconds. + +--- + +# View Dashboard + +Serve the project locally using any static web server. + +Example: + +```bash python -m http.server 8000 ``` -افتح المتصفح على: `http://localhost:8000/dashboard.html` -> ملاحظة: الـ dashboard يقرأ `history.json` و`latest.json` عبر `fetch()`، لذلك -> يجب تشغيله من خادم HTTP (مثل `http.server`) وليس بفتح الملف مباشرة (`file://`) -> لتفادي قيود CORS في المتصفح. +Then open: -## 2) النشر على Ghaymah - -نفس نمط النشر في المهمة 1: نضع `collector.py` و`dashboard.html` خلف سيرفر -بسيط (مثال: Flask/`http.server`) داخل حاوية واحدة، وننشرها كتطبيق Ghaymah: - -```dockerfile -FROM python:3.11-slim -WORKDIR /app -COPY requirements.txt . -RUN pip install --no-cache-dir -r requirements.txt -COPY collector.py dashboard.html ./ -EXPOSE 8000 -# يشغّل التجميع في الخلفية + خادم استضافة الملفات في الواجهة -CMD sh -c "python collector.py & python -m http.server 8000" ``` +http://localhost:8000/dashboard.html +``` + +--- + +# Docker + +Build the image: ```bash -docker build -t mithal-monitor:v1 . -docker push registry.ghaymah.systems//mithal-monitor:v1 -ghaymah deploy --app mithal-monitor \ - --image registry.ghaymah.systems//mithal-monitor:v1 \ - --port 8000 +docker build -t mithal-dashboard . ``` -بعد النشر، الرابط الناتج (مثال: `https://mithal-monitor.ghaymah.systems/dashboard.html`) -يعرض اللوحة الحية، وتُحدَّث القراءات تلقائياً كل دقيقة عبر `collector.py` -الذي يعمل باستمرار داخل نفس الحاوية. +Run the container: -## 3) ملاحظة حول رابط البحث (Search Response) -عدّل المتغير `SEARCH_URL` داخل `collector.py` ليطابق مسار البحث الفعلي في -موقع mithal.space (حالياً موضوع كمثال `mithal.space/search?q=test`)، حسب -البنية الفعلية للموقع. +```bash +docker run -p 8080:8080 mithal-dashboard +``` + +--- + +# Deployment + +The dashboard can be deployed to **Ghaymah Cloud** using the official CLI. + +```bash +gy auth login +``` + +Initialize the application: + +```bash +gy resource app init --project-id +``` + +Deploy: + +```bash +gy resource app launch +``` + +--- + +# Dashboard Preview + +The dashboard displays: + +* Overall availability +* Website health +* SSL certificate information +* DNS performance +* Response latency graph +* Search endpoint performance +* Latest monitoring history + +--- + +# Future Improvements + +* Email notifications +* Slack alerts +* Telegram integration +* Prometheus exporter +* Grafana integration +* Multi-site monitoring +* Error trend analysis +* Response time statistics +* Dark/Light theme switch +* Authentication for dashboard access + +--- + +# Author + +**Moamen Lotfy** + +DevOps Engineer + +Ghaymah Cloud Training Project – Task 5 diff --git a/task5-mithal-dashboard/collector.py b/task5-mithal-dashboard/collector.py index 12ef11a..3475fef 100644 --- a/task5-mithal-dashboard/collector.py +++ b/task5-mithal-dashboard/collector.py @@ -1,17 +1,3 @@ -""" -collector.py -يجمع مقاييس مراقبة لموقع mithal.space كل دقيقة ويخزنها في CSV و JSON: - - Latency: زمن استجابة HTTP - - Uptime: هل الموقع متاح (status code) - - SSL: حالة الشهادة وتاريخ انتهائها - - DNS: وقت تحليل DNS - - Search Response: زمن الرد على عملية بحث (query) داخل الموقع - -الاستخدام: - python collector.py # يعمل باستمرار كل 60 ثانية - python collector.py --once # فحص واحد فقط (مفيد للاختبار / cron) -""" - import argparse import csv import json diff --git a/task5-mithal-dashboard/history.json b/task5-mithal-dashboard/history.json new file mode 100644 index 0000000..af527c2 --- /dev/null +++ b/task5-mithal-dashboard/history.json @@ -0,0 +1,12 @@ +[ + { + "timestamp": "2026-07-28T20:05:08.595660+00:00", + "up": true, + "status_code": 200, + "latency_ms": 954.67, + "dns_ms": 1.3, + "ssl_valid": true, + "ssl_days_left": 48, + "search_latency_ms": 832.81 + } +] \ No newline at end of file diff --git a/task5-mithal-dashboard/latest.json b/task5-mithal-dashboard/latest.json new file mode 100644 index 0000000..c32b9f6 --- /dev/null +++ b/task5-mithal-dashboard/latest.json @@ -0,0 +1,10 @@ +{ + "timestamp": "2026-07-28T20:05:08.595660+00:00", + "up": true, + "status_code": 200, + "latency_ms": 954.67, + "dns_ms": 1.3, + "ssl_valid": true, + "ssl_days_left": 48, + "search_latency_ms": 832.81 +} \ No newline at end of file diff --git a/task5-mithal-dashboard/metrics.csv b/task5-mithal-dashboard/metrics.csv new file mode 100644 index 0000000..145c7dd --- /dev/null +++ b/task5-mithal-dashboard/metrics.csv @@ -0,0 +1,2 @@ +timestamp,up,status_code,latency_ms,dns_ms,ssl_valid,ssl_days_left,search_latency_ms +2026-07-28T20:05:08.595660+00:00,True,200,954.67,1.3,True,48,832.81