Complete interview solution
هذا الالتزام موجود في:
@@ -0,0 +1,213 @@
|
||||
# Incident Postmortem Report
|
||||
|
||||
## Application Information
|
||||
|
||||
| Item | Value |
|
||||
|------|-------|
|
||||
| **Application** | API Service on Ghaymah Platform |
|
||||
| **Incident Date** | 2026-07-27 |
|
||||
| **Incident Duration** | 45 Minutes (10:15 – 11:00) |
|
||||
| **Severity** | High |
|
||||
| **Impact** | Complete service outage. Users were unable to access the API during the incident. |
|
||||
|
||||
---
|
||||
|
||||
# 1. Executive Summary
|
||||
|
||||
On **2026-07-27**, the API service experienced a **45-minute outage** due to repeated **OOMKilled** events.
|
||||
|
||||
The application gradually consumed more memory because of a memory leak while processing large requests. Once the container exceeded its allocated memory limit, the platform terminated and restarted it automatically. This restart cycle repeated continuously, resulting in complete service unavailability.
|
||||
|
||||
The immediate mitigation was increasing the container memory allocation and restarting the service. Long-term corrective actions include fixing the memory leak, improving monitoring, and implementing an auto-scaling policy based on resource utilization.
|
||||
|
||||
---
|
||||
|
||||
# 2. Incident Timeline
|
||||
|
||||
| Time | Event |
|
||||
|------|-------|
|
||||
| **10:15** | Memory usage started increasing on one of the application containers. |
|
||||
| **10:22** | First container was terminated with **OOMKilled** and automatically restarted by the platform. |
|
||||
| **10:25 – 10:45** | Multiple containers repeatedly entered restart loops, reducing the application's availability. |
|
||||
| **10:45** | Monitoring system generated alerts indicating complete service outage. |
|
||||
| **10:50** | SRE team began investigating the incident and identified abnormal memory usage. |
|
||||
| **11:00** | Container memory limit increased from **512 MiB** to **1 GiB** and services were restarted successfully. |
|
||||
| **11:15** | Additional monitoring and temporary mitigation measures were applied. |
|
||||
|
||||
---
|
||||
|
||||
# 3. Root Cause Analysis
|
||||
|
||||
## Immediate Cause
|
||||
|
||||
The application exhausted the available memory, causing the platform to terminate the container with an **OOMKilled** event.
|
||||
|
||||
## Root Cause
|
||||
|
||||
A memory leak inside the request processing logic (or an external library) continuously retained memory instead of releasing it after requests were completed.
|
||||
|
||||
## Contributing Factors
|
||||
|
||||
- Sudden increase in traffic (approximately **3× normal load**).
|
||||
- Insufficient memory allocation.
|
||||
- No automatic scaling based on memory utilization.
|
||||
- Lack of early warning alerts before reaching the memory limit.
|
||||
|
||||
---
|
||||
|
||||
# 4. Recommendations
|
||||
|
||||
## Short-Term Actions
|
||||
|
||||
- Increase container memory limits.
|
||||
- Monitor memory consumption continuously.
|
||||
- Restart unhealthy containers automatically.
|
||||
|
||||
## Long-Term Improvements
|
||||
|
||||
- Fix the memory leak in the application.
|
||||
- Configure appropriate container memory **requests** and **limits**.
|
||||
- Enable memory-based auto-scaling.
|
||||
- Configure proactive alerting.
|
||||
- Perform regular load and stress testing.
|
||||
- Maintain an incident response runbook for OOMKilled events.
|
||||
|
||||
---
|
||||
|
||||
# 5. Auto-Scaling Policy
|
||||
|
||||
To prevent similar incidents, the following auto-scaling policy is recommended.
|
||||
|
||||
| Configuration | Value |
|
||||
|--------------|-------|
|
||||
| **Metric** | Average Container Memory Usage |
|
||||
| **Scale-Out Threshold** | Memory usage > **70%** for **2 minutes** |
|
||||
| **Scale-Out Action** | Add one container instance |
|
||||
| **Minimum Replicas** | 2 |
|
||||
| **Maximum Replicas** | 10 |
|
||||
| **Cooldown Period** | 5 Minutes |
|
||||
| **Scale-In Threshold** | Memory usage < **40%** for **10 minutes** |
|
||||
| **Scale-In Action** | Remove one container instance |
|
||||
|
||||
### Example Auto-Scaling Configuration
|
||||
|
||||
```yaml
|
||||
apiVersion: autoscaling/v2
|
||||
kind: HorizontalPodAutoscaler
|
||||
metadata:
|
||||
name: api-hpa
|
||||
spec:
|
||||
minReplicas: 2
|
||||
maxReplicas: 10
|
||||
metrics:
|
||||
- type: Resource
|
||||
resource:
|
||||
name: memory
|
||||
target:
|
||||
type: Utilization
|
||||
averageUtilization: 70
|
||||
```
|
||||
|
||||
> **Note:** The YAML above is an example of a memory-based auto-scaling policy and illustrates the desired behavior conceptually.
|
||||
|
||||
---
|
||||
|
||||
# 6. Early Detection Using Ghaymah Monitoring
|
||||
|
||||
The following monitoring capabilities should be configured to detect similar incidents before they impact users.
|
||||
|
||||
## Resource Monitoring
|
||||
|
||||
Track real-time metrics including:
|
||||
|
||||
- Container Memory Usage
|
||||
- CPU Utilization
|
||||
- Container Restart Count
|
||||
- Container Health Status
|
||||
|
||||
---
|
||||
|
||||
## Alerting
|
||||
|
||||
Configure alerts such as:
|
||||
|
||||
| Condition | Action |
|
||||
|-----------|--------|
|
||||
| Memory usage > 60% for 3 minutes | Send Email / Slack / Webhook notification |
|
||||
| Multiple container restarts | Trigger High-Priority Alert |
|
||||
| Service unavailable | Immediate Critical Alert |
|
||||
|
||||
---
|
||||
|
||||
## Event Monitoring
|
||||
|
||||
Monitor platform events including:
|
||||
|
||||
- OOMKilled
|
||||
- Container Restart
|
||||
- CrashLoop
|
||||
- Failed Deployments
|
||||
|
||||
---
|
||||
|
||||
## Dashboard
|
||||
|
||||
Create a monitoring dashboard displaying:
|
||||
|
||||
- Current Container Status
|
||||
- Memory Utilization
|
||||
- CPU Utilization
|
||||
- Response Time
|
||||
- Request Count
|
||||
- Restart Count
|
||||
- OOMKilled Events
|
||||
- Service Availability (Uptime)
|
||||
|
||||
---
|
||||
|
||||
## Trend Analysis
|
||||
|
||||
Analyze historical metrics to identify:
|
||||
|
||||
- Gradual memory growth
|
||||
- Traffic spikes
|
||||
- Abnormal restart frequency
|
||||
- Resource utilization trends
|
||||
|
||||
This enables proactive detection before a service outage occurs.
|
||||
|
||||
---
|
||||
|
||||
# 7. Lessons Learned
|
||||
|
||||
The incident highlighted several important operational improvements:
|
||||
|
||||
- Proper memory sizing is essential.
|
||||
- Continuous monitoring should be proactive rather than reactive.
|
||||
- Memory-based auto-scaling reduces service interruption.
|
||||
- Early alerts significantly reduce recovery time.
|
||||
- Load testing helps identify memory leaks before production deployment.
|
||||
|
||||
---
|
||||
|
||||
# Action Items
|
||||
|
||||
| Priority | Action | Status |
|
||||
|----------|--------|--------|
|
||||
| High | Fix memory leak | Planned |
|
||||
| High | Configure memory alerts | Planned |
|
||||
| High | Enable auto-scaling | Planned |
|
||||
| Medium | Perform load testing | Planned |
|
||||
| Medium | Update operational runbook | Planned |
|
||||
|
||||
---
|
||||
|
||||
# Report Information
|
||||
|
||||
| Item | Value |
|
||||
|------|-------|
|
||||
| **Prepared By** | SRE Team |
|
||||
| **Platform** | Ghaymah Systems |
|
||||
| **Report Date** | 2026-07-27 |
|
||||
| **Incident Type** | Repeated OOMKilled |
|
||||
| **Status** | Closed |
|
||||
235
q3-cicd/README.md
Normal file
235
q3-cicd/README.md
Normal file
@@ -0,0 +1,235 @@
|
||||
# CI/CD Pipeline Documentation
|
||||
|
||||
This document describes the Continuous Integration and Continuous Deployment (CI/CD) pipeline implemented for the Ghaymah SRE exam.
|
||||
|
||||
---
|
||||
|
||||
# Pipeline Overview
|
||||
|
||||
The pipeline automates the application delivery process by:
|
||||
|
||||
1. Building a Docker image.
|
||||
2. Pushing the image to the Ghaymah Container Registry.
|
||||
3. Deploying automatically to the **Staging** environment.
|
||||
4. Waiting for **manual approval**.
|
||||
5. Deploying the same image to **Production**.
|
||||
|
||||
```
|
||||
Developer Push
|
||||
│
|
||||
▼
|
||||
GitHub Actions
|
||||
│
|
||||
▼
|
||||
Build Docker Image
|
||||
│
|
||||
▼
|
||||
Push to Ghaymah Container Registry
|
||||
│
|
||||
▼
|
||||
Deploy to Staging
|
||||
│
|
||||
▼
|
||||
Manual Approval
|
||||
│
|
||||
▼
|
||||
Deploy to Production
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
# Manual Approval for Production
|
||||
|
||||
Production deployments are protected using **GitHub Environments**.
|
||||
|
||||
## Configuration
|
||||
|
||||
1. Open the repository.
|
||||
|
||||
2. Navigate to:
|
||||
|
||||
```
|
||||
Settings
|
||||
→ Environments
|
||||
```
|
||||
|
||||
3. Create an environment named:
|
||||
|
||||
```
|
||||
production
|
||||
```
|
||||
|
||||
4. Enable:
|
||||
|
||||
- Required reviewers
|
||||
- (Optional) Wait timer
|
||||
- (Optional) Deployment branch restrictions
|
||||
|
||||
---
|
||||
|
||||
## Deployment Flow
|
||||
|
||||
After the application is successfully deployed to **Staging**, the workflow pauses before deploying to **Production**.
|
||||
|
||||
```
|
||||
Build
|
||||
│
|
||||
Deploy Staging
|
||||
│
|
||||
Waiting for Approval
|
||||
│
|
||||
Reviewer Approves
|
||||
│
|
||||
Deploy Production
|
||||
```
|
||||
|
||||
Only authorized reviewers can approve the deployment.
|
||||
|
||||
---
|
||||
|
||||
## Benefits
|
||||
|
||||
- Prevents accidental deployments.
|
||||
- Adds an approval gate before production.
|
||||
- Provides a complete audit trail.
|
||||
- Supports controlled production releases.
|
||||
|
||||
---
|
||||
|
||||
# Staging vs Production
|
||||
|
||||
| Feature | Staging | Production |
|
||||
|----------|----------|------------|
|
||||
| Purpose | Integration testing and QA | Live customer environment |
|
||||
| Users | Developers & QA | End users |
|
||||
| Data | Test or sanitized data | Real production data |
|
||||
| Replicas | Usually 1 | Usually 3 or more |
|
||||
| Resources | Lower CPU/Memory | Higher CPU/Memory |
|
||||
| Secrets | Test credentials | Production credentials |
|
||||
| Deployment | Automatic | Manual approval required |
|
||||
| Monitoring | Basic monitoring | Full monitoring and alerting |
|
||||
| Rollback | Low impact | Carefully managed |
|
||||
| Domain | staging.example.com | api.example.com |
|
||||
| Logging | Verbose | Optimized for production |
|
||||
|
||||
> **Best Practice**
|
||||
>
|
||||
> The **same Docker image** (identified by its commit SHA) should be promoted from **Staging** to **Production**. Rebuilding the image for Production should be avoided to guarantee consistency.
|
||||
|
||||
---
|
||||
|
||||
# Ghaymah CLI Integration
|
||||
|
||||
> **Note**
|
||||
>
|
||||
> The official Ghaymah CLI documentation was not provided with the exam. The following commands are **placeholders** and should be replaced with the official syntax when available.
|
||||
|
||||
---
|
||||
|
||||
## 1. Install CLI
|
||||
|
||||
```bash
|
||||
curl -sSL https://get.ghaymah.systems/cli/install.sh | sh
|
||||
|
||||
export PATH="$HOME/.ghaymah/bin:$PATH"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 2. Authenticate
|
||||
|
||||
```bash
|
||||
ghaymah login --token $GHAYMAH_TOKEN
|
||||
```
|
||||
|
||||
Authentication credentials should be stored securely using GitHub Secrets.
|
||||
|
||||
---
|
||||
|
||||
## 3. Deploy Application
|
||||
|
||||
```bash
|
||||
ghaymah deploy \
|
||||
--image registry.ghaymah.systems/ghaymah-api:latest \
|
||||
--name api-production \
|
||||
--port 5000 \
|
||||
--env FLASK_ENV=production \
|
||||
--replicas 3 \
|
||||
--health-check /health
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 4. Verify Deployment
|
||||
|
||||
```bash
|
||||
ghaymah status --name api-production
|
||||
|
||||
ghaymah logs --name api-production
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 5. Scale Application
|
||||
|
||||
```bash
|
||||
ghaymah scale \
|
||||
--name api-production \
|
||||
--replicas 5
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 6. Rollback
|
||||
|
||||
```bash
|
||||
ghaymah deploy \
|
||||
--image registry.ghaymah.systems/ghaymah-api:<previous-tag> \
|
||||
--name api-production \
|
||||
--port 5000
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
# GitHub Secrets
|
||||
|
||||
Sensitive information should **never** be stored in the repository.
|
||||
|
||||
Configure the following repository secrets:
|
||||
|
||||
| Secret | Description |
|
||||
|---------|-------------|
|
||||
| `GHAYMAH_USERNAME` | Container Registry username |
|
||||
| `GHAYMAH_TOKEN` | Access token used for authentication |
|
||||
|
||||
These secrets are referenced in the workflow as:
|
||||
|
||||
```yaml
|
||||
${{ secrets.GHAYMAH_USERNAME }}
|
||||
|
||||
${{ secrets.GHAYMAH_TOKEN }}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
# Best Practices
|
||||
|
||||
- Build the Docker image once and promote the same image between environments.
|
||||
- Tag every image with the Git commit SHA for traceability.
|
||||
- Keep secrets in **GitHub Secrets**, never in source code.
|
||||
- Protect Production using **GitHub Environment approvals**.
|
||||
- Enable Docker Buildx cache to reduce build time.
|
||||
- Use health checks before considering a deployment successful.
|
||||
- Maintain separate configurations for Staging and Production.
|
||||
- Document deployment and rollback procedures.
|
||||
- Monitor deployments and configure alerting for failures.
|
||||
|
||||
---
|
||||
|
||||
# References
|
||||
|
||||
- GitHub Actions Documentation
|
||||
- Docker Buildx Documentation
|
||||
- GitHub Environments Documentation
|
||||
|
||||
> Ghaymah CLI commands in this document are provided as implementation placeholders because the official CLI documentation was not included with the exam.
|
||||
@@ -0,0 +1,136 @@
|
||||
name: Build and Deploy to Ghaymah
|
||||
|
||||
on:
|
||||
# Run automatically when code is pushed to the main branch
|
||||
push:
|
||||
branches:
|
||||
- main
|
||||
|
||||
# Allow manual workflow execution from GitHub Actions
|
||||
workflow_dispatch:
|
||||
|
||||
env:
|
||||
# Replace with the official Ghaymah Container Registry URL
|
||||
REGISTRY: registry.ghaymah.systems
|
||||
|
||||
# Docker image name
|
||||
IMAGE_NAME: ghaymah-api
|
||||
|
||||
jobs:
|
||||
|
||||
# -------------------------------------------------------
|
||||
# Build the Docker image and push it to the registry
|
||||
# -------------------------------------------------------
|
||||
build-and-push:
|
||||
runs-on: ubuntu-latest
|
||||
|
||||
permissions:
|
||||
contents: read
|
||||
packages: write
|
||||
|
||||
steps:
|
||||
# Check out the repository source code
|
||||
- name: Checkout code
|
||||
uses: actions/checkout@v4
|
||||
|
||||
# Enable Docker Buildx for advanced builds and caching
|
||||
- name: Set up Docker Buildx
|
||||
uses: docker/setup-buildx-action@v3
|
||||
|
||||
# Authenticate with the container registry
|
||||
- name: Log in to Ghaymah Container Registry
|
||||
uses: docker/login-action@v3
|
||||
with:
|
||||
registry: ${{ env.REGISTRY }}
|
||||
username: ${{ secrets.GHAYMAH_USERNAME }}
|
||||
password: ${{ secrets.GHAYMAH_TOKEN }}
|
||||
|
||||
# Build the Docker image and push it to the registry
|
||||
- name: Build and push Docker image
|
||||
uses: docker/build-push-action@v6
|
||||
with:
|
||||
# Path containing the Dockerfile
|
||||
context: ./q1-deploy-monitor
|
||||
|
||||
# Push the image after a successful build
|
||||
push: true
|
||||
|
||||
# Publish two image tags:
|
||||
# - latest
|
||||
# - commit SHA for traceability
|
||||
tags: |
|
||||
${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}:latest
|
||||
${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}:${{ github.sha }}
|
||||
|
||||
# Enable Docker layer caching
|
||||
cache-from: type=gha
|
||||
cache-to: type=gha,mode=max
|
||||
|
||||
# -------------------------------------------------------
|
||||
# Deploy the application to the staging environment
|
||||
# -------------------------------------------------------
|
||||
deploy-staging:
|
||||
needs: build-and-push
|
||||
runs-on: ubuntu-latest
|
||||
|
||||
# GitHub Environment
|
||||
environment: staging
|
||||
|
||||
steps:
|
||||
# Install the Ghaymah CLI
|
||||
# Replace with the official installation command if different
|
||||
- name: Install Ghaymah CLI
|
||||
run: |
|
||||
curl -sSL https://get.ghaymah.systems/cli/install.sh | sh
|
||||
echo "$HOME/.ghaymah/bin" >> $GITHUB_PATH
|
||||
|
||||
# Deploy the application to staging
|
||||
# Replace the commands below with the official CLI syntax if needed
|
||||
- name: Deploy to Staging
|
||||
env:
|
||||
GHAYMAH_TOKEN: ${{ secrets.GHAYMAH_TOKEN }}
|
||||
|
||||
run: |
|
||||
ghaymah login --token $GHAYMAH_TOKEN
|
||||
|
||||
ghaymah deploy \
|
||||
--image ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}:${{ github.sha }} \
|
||||
--name api-staging \
|
||||
--port 5000 \
|
||||
--env FLASK_ENV=staging \
|
||||
--replicas 1
|
||||
|
||||
# -------------------------------------------------------
|
||||
# Deploy the application to the production environment
|
||||
# -------------------------------------------------------
|
||||
deploy-production:
|
||||
needs: deploy-staging
|
||||
runs-on: ubuntu-latest
|
||||
|
||||
# Configure this environment with required reviewers
|
||||
# to enable manual approval before deployment
|
||||
environment: production
|
||||
|
||||
steps:
|
||||
# Install the Ghaymah CLI
|
||||
- name: Install Ghaymah CLI
|
||||
run: |
|
||||
curl -sSL https://get.ghaymah.systems/cli/install.sh | sh
|
||||
echo "$HOME/.ghaymah/bin" >> $GITHUB_PATH
|
||||
|
||||
# Deploy the application to production
|
||||
# This job runs only after manual approval
|
||||
- name: Deploy to Production
|
||||
env:
|
||||
GHAYMAH_TOKEN: ${{ secrets.GHAYMAH_TOKEN }}
|
||||
|
||||
run: |
|
||||
ghaymah login --token $GHAYMAH_TOKEN
|
||||
|
||||
ghaymah deploy \
|
||||
--image ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}:${{ github.sha }} \
|
||||
--name api-production \
|
||||
--port 5000 \
|
||||
--env FLASK_ENV=production \
|
||||
--replicas 3 \
|
||||
--health-check /health
|
||||
ملف ثنائي غير معروض.
|
قبل العرض: | الارتفاع: | الحجم: 0 B بعد العرض: | الارتفاع: | الحجم: 146 KiB |
@@ -0,0 +1,167 @@
|
||||
# Scalability and Load Balancing Design
|
||||
|
||||
## Scenario
|
||||
|
||||
Design an architecture capable of handling:
|
||||
|
||||
- **15,000 requests per second (req/s)**
|
||||
|
||||
Each application container can process:
|
||||
|
||||
- **500 req/s**
|
||||
|
||||
A **30% capacity buffer** is required to handle traffic spikes and unexpected load.
|
||||
|
||||
---
|
||||
|
||||
# 1. High-Level Architecture
|
||||
|
||||
The proposed architecture consists of the following components:
|
||||
|
||||
- DNS
|
||||
- Load Balancer
|
||||
- Multiple API containers
|
||||
- Auto Scaling service
|
||||
- Container Orchestrator
|
||||
- Database
|
||||
- Ghaymah Block Storage
|
||||
- Monitoring and Alerting
|
||||
|
||||
## Architecture Diagram
|
||||
|
||||
The following diagram illustrates the proposed scalable architecture for handling 15,000 requests per second using load balancing, auto scaling, centralized monitoring, and persistent storage.
|
||||
|
||||
|
||||
|
||||
---
|
||||
|
||||
## Request Flow
|
||||
|
||||
```
|
||||
Users
|
||||
│
|
||||
▼
|
||||
DNS
|
||||
│
|
||||
▼
|
||||
Load Balancer
|
||||
│
|
||||
▼
|
||||
API Containers
|
||||
│
|
||||
▼
|
||||
Database
|
||||
```
|
||||
|
||||
Monitoring components (Prometheus, Grafana, and centralized logging) collect metrics and logs from the Load Balancer, API containers, and database.
|
||||
|
||||
---
|
||||
|
||||
# 2. Container Capacity Calculation
|
||||
|
||||
## Step 1: Calculate Required Containers
|
||||
|
||||
Traffic:
|
||||
|
||||
```
|
||||
15,000 req/s
|
||||
```
|
||||
|
||||
Container capacity:
|
||||
|
||||
```
|
||||
500 req/s
|
||||
```
|
||||
|
||||
Required containers:
|
||||
|
||||
```
|
||||
15000 / 500 = 30 containers
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Step 2: Add 30% Safety Buffer
|
||||
|
||||
```
|
||||
30 × 1.30 = 39 containers
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Recommended Deployment Size
|
||||
|
||||
Minimum recommended deployment:
|
||||
|
||||
**39 containers**
|
||||
|
||||
This provides additional capacity for traffic spikes and reduces the risk of resource saturation during unexpected load increases.
|
||||
|
||||
---
|
||||
|
||||
# 3. Cold Start Strategy
|
||||
|
||||
To minimize startup delays during scaling events, the following strategies are recommended.
|
||||
|
||||
## Warm Pool
|
||||
|
||||
Maintain 2–3 idle containers ready to receive traffic immediately.
|
||||
|
||||
## Container Image Optimization
|
||||
|
||||
- Use lightweight base images.
|
||||
- Remove unnecessary packages and dependencies.
|
||||
- Keep image size as small as possible.
|
||||
|
||||
## Image Pre-Pulling
|
||||
|
||||
Pre-pull container images on worker nodes to reduce deployment time and avoid downloading images during scale-out events.
|
||||
|
||||
## Health Checks
|
||||
|
||||
New containers must pass readiness and liveness checks before receiving production traffic.
|
||||
|
||||
## Gradual Traffic Shift
|
||||
|
||||
The Load Balancer should gradually route traffic to newly started containers after successful health verification.
|
||||
|
||||
## Predictive Scaling
|
||||
|
||||
Scale proactively based on CPU utilization, memory usage, and request-rate trends instead of waiting until the system reaches full capacity.
|
||||
|
||||
---
|
||||
|
||||
# 4. Using Ghaymah Block Storage for Stateful Workloads
|
||||
|
||||
Ghaymah Block Storage provides persistent storage volumes that remain available even if containers are restarted, replaced, or rescheduled.
|
||||
|
||||
This makes it suitable for workloads that require durable data storage.
|
||||
|
||||
Examples include:
|
||||
|
||||
- Databases
|
||||
- Message queues
|
||||
- File storage
|
||||
- Application uploads
|
||||
- Persistent logs
|
||||
|
||||
## Benefits
|
||||
|
||||
- Persistent data survives container restarts.
|
||||
- Volumes can be attached to replacement containers.
|
||||
- Improves reliability for stateful applications.
|
||||
- Simplifies backup and disaster recovery.
|
||||
- Enhances data durability during scaling events.
|
||||
|
||||
---
|
||||
|
||||
# Best Practices
|
||||
|
||||
- Deploy multiple replicas behind a Load Balancer.
|
||||
- Enable Horizontal Auto Scaling based on CPU, memory, and request rate.
|
||||
- Configure readiness and liveness probes.
|
||||
- Use lightweight container images to reduce cold-start time.
|
||||
- Store persistent data using Ghaymah Block Storage.
|
||||
- Continuously monitor latency, error rate, CPU, and memory utilization.
|
||||
- Configure alerts for abnormal response times and infrastructure failures.
|
||||
- Regularly perform load testing to validate scalability assumptions.
|
||||
@@ -0,0 +1,336 @@
|
||||
<!DOCTYPE html>
|
||||
<html lang="en">
|
||||
<head>
|
||||
<meta charset="UTF-8">
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1.0">
|
||||
<title>mithal.space Monitoring Dashboard</title>
|
||||
<script src="https://cdn.jsdelivr.net/npm/chart.js@4.4.0/dist/chart.umd.min.js"></script>
|
||||
<style>
|
||||
* {
|
||||
box-sizing: border-box;
|
||||
margin: 0;
|
||||
padding: 0;
|
||||
}
|
||||
body {
|
||||
font-family: 'Segoe UI', Tahoma, Geneva, Verdana, sans-serif;
|
||||
background: #f0f2f5;
|
||||
padding: 20px;
|
||||
}
|
||||
.container {
|
||||
max-width: 1400px;
|
||||
margin: 0 auto;
|
||||
}
|
||||
.header {
|
||||
background: linear-gradient(135deg, #667eea 0%, #764ba2 100%);
|
||||
color: white;
|
||||
padding: 20px 30px;
|
||||
border-radius: 12px;
|
||||
margin-bottom: 20px;
|
||||
}
|
||||
.header h1 {
|
||||
font-weight: 300;
|
||||
font-size: 2rem;
|
||||
}
|
||||
.header .subtitle {
|
||||
opacity: 0.9;
|
||||
margin-top: 5px;
|
||||
}
|
||||
.grid {
|
||||
display: grid;
|
||||
grid-template-columns: repeat(auto-fit, minmax(220px, 1fr));
|
||||
gap: 20px;
|
||||
margin-bottom: 20px;
|
||||
}
|
||||
.card {
|
||||
background: white;
|
||||
padding: 20px;
|
||||
border-radius: 12px;
|
||||
box-shadow: 0 2px 8px rgba(0,0,0,0.08);
|
||||
transition: transform 0.2s;
|
||||
}
|
||||
.card:hover {
|
||||
transform: translateY(-3px);
|
||||
box-shadow: 0 4px 16px rgba(0,0,0,0.12);
|
||||
}
|
||||
.card-label {
|
||||
font-size: 0.85rem;
|
||||
text-transform: uppercase;
|
||||
letter-spacing: 0.5px;
|
||||
color: #6c757d;
|
||||
margin-bottom: 8px;
|
||||
}
|
||||
.card-value {
|
||||
font-size: 2rem;
|
||||
font-weight: 600;
|
||||
}
|
||||
.card-value .unit {
|
||||
font-size: 1rem;
|
||||
font-weight: 400;
|
||||
color: #6c757d;
|
||||
margin-left: 4px;
|
||||
}
|
||||
.status-up { color: #28a745; }
|
||||
.status-down { color: #dc3545; }
|
||||
.chart-container {
|
||||
background: white;
|
||||
padding: 20px;
|
||||
border-radius: 12px;
|
||||
box-shadow: 0 2px 8px rgba(0,0,0,0.08);
|
||||
margin-bottom: 20px;
|
||||
height: 300px;
|
||||
}
|
||||
.table-container {
|
||||
background: white;
|
||||
padding: 20px;
|
||||
border-radius: 12px;
|
||||
box-shadow: 0 2px 8px rgba(0,0,0,0.08);
|
||||
overflow-x: auto;
|
||||
}
|
||||
table {
|
||||
width: 100%;
|
||||
border-collapse: collapse;
|
||||
font-size: 0.9rem;
|
||||
}
|
||||
th {
|
||||
text-align: left;
|
||||
padding: 12px 8px;
|
||||
border-bottom: 2px solid #dee2e6;
|
||||
color: #495057;
|
||||
font-weight: 600;
|
||||
}
|
||||
td {
|
||||
padding: 10px 8px;
|
||||
border-bottom: 1px solid #e9ecef;
|
||||
}
|
||||
.badge {
|
||||
display: inline-block;
|
||||
padding: 2px 12px;
|
||||
border-radius: 20px;
|
||||
font-size: 0.8rem;
|
||||
font-weight: 600;
|
||||
}
|
||||
.badge-up { background: #d4edda; color: #155724; }
|
||||
.badge-down { background: #f8d7da; color: #721c24; }
|
||||
.badge-ssl-ok { background: #d4edda; color: #155724; }
|
||||
.badge-ssl-warning { background: #fff3cd; color: #856404; }
|
||||
.badge-ssl-danger { background: #f8d7da; color: #721c24; }
|
||||
.text-muted { color: #6c757d; font-size: 0.85rem; }
|
||||
.text-center { text-align: center; }
|
||||
.last-update {
|
||||
margin-top: 20px;
|
||||
padding: 12px;
|
||||
background: white;
|
||||
border-radius: 8px;
|
||||
text-align: center;
|
||||
color: #6c757d;
|
||||
font-size: 0.9rem;
|
||||
}
|
||||
@media (max-width: 600px) {
|
||||
.grid { grid-template-columns: 1fr 1fr; }
|
||||
.card-value { font-size: 1.5rem; }
|
||||
}
|
||||
</style>
|
||||
</head>
|
||||
<body>
|
||||
<div class="container">
|
||||
<div class="header">
|
||||
<h1>📊 mithal.space Monitoring Dashboard</h1>
|
||||
<div class="subtitle">Real-time performance & availability monitoring</div>
|
||||
</div>
|
||||
|
||||
<div class="grid">
|
||||
<div class="card">
|
||||
<div class="card-label">🟢 Uptime (24h)</div>
|
||||
<div class="card-value" id="uptime">--</div>
|
||||
</div>
|
||||
<div class="card">
|
||||
<div class="card-label">📡 Current Status</div>
|
||||
<div class="card-value" id="status">--</div>
|
||||
</div>
|
||||
<div class="card">
|
||||
<div class="card-label">⚡ Last Latency</div>
|
||||
<div class="card-value" id="last-latency">-- <span class="unit">ms</span></div>
|
||||
</div>
|
||||
<div class="card">
|
||||
<div class="card-label">🔒 SSL Certificate</div>
|
||||
<div class="card-value" id="ssl-status">--</div>
|
||||
<div class="text-muted" id="ssl-days">-- days remaining</div>
|
||||
</div>
|
||||
<div class="card">
|
||||
<div class="card-label">🌐 DNS Resolution</div>
|
||||
<div class="card-value" id="dns-latency">-- <span class="unit">ms</span></div>
|
||||
</div>
|
||||
<div class="card">
|
||||
<div class="card-label">🔍 Search Response</div>
|
||||
<div class="card-value" id="search-latency">-- <span class="unit">ms</span></div>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<div class="chart-container">
|
||||
<canvas id="latencyChart"></canvas>
|
||||
</div>
|
||||
|
||||
<div class="table-container">
|
||||
<h3 style="margin-bottom: 15px; font-weight: 400;">📋 Recent Checks (last 10)</h3>
|
||||
<table>
|
||||
<thead>
|
||||
<tr>
|
||||
<th>Timestamp</th>
|
||||
<th>Status</th>
|
||||
<th>Latency (ms)</th>
|
||||
<th>DNS (ms)</th>
|
||||
<th>SSL</th>
|
||||
<th>Search (ms)</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody id="checks-body">
|
||||
<tr><td colspan="6" class="text-muted text-center">Loading data...</td></tr>
|
||||
</tbody>
|
||||
</table>
|
||||
</div>
|
||||
|
||||
<div class="last-update" id="last-update">📡 Last updated: Loading...</div>
|
||||
</div>
|
||||
|
||||
<script>
|
||||
let chart = null;
|
||||
|
||||
function computeUptime(data) {
|
||||
if (!data || data.length === 0) return 0;
|
||||
const now = new Date();
|
||||
const last24h = data.filter(entry => {
|
||||
const ts = new Date(entry.timestamp);
|
||||
return (now - ts) <= 24 * 60 * 60 * 1000;
|
||||
});
|
||||
if (last24h.length === 0) return 0;
|
||||
const up = last24h.filter(e => e.is_up === true).length;
|
||||
return (up / last24h.length) * 100;
|
||||
}
|
||||
|
||||
function getSslBadge(days) {
|
||||
if (days === null || days === undefined)
|
||||
return '<span class="badge badge-down">❌ Unknown</span>';
|
||||
if (days > 30)
|
||||
return '<span class="badge badge-ssl-ok">✅ Valid</span>';
|
||||
else if (days > 7)
|
||||
return '<span class="badge badge-ssl-warning">⚠️ Expiring soon</span>';
|
||||
else
|
||||
return '<span class="badge badge-ssl-danger">🚨 Expiring!</span>';
|
||||
}
|
||||
|
||||
function updateDashboard(data) {
|
||||
if (!data || data.length === 0) {
|
||||
document.getElementById('uptime').textContent = '--';
|
||||
document.getElementById('status').innerHTML = '--';
|
||||
document.getElementById('last-latency').innerHTML = '-- <span class="unit">ms</span>';
|
||||
document.getElementById('ssl-status').textContent = '--';
|
||||
document.getElementById('ssl-days').textContent = '-- days remaining';
|
||||
document.getElementById('dns-latency').innerHTML = '-- <span class="unit">ms</span>';
|
||||
document.getElementById('search-latency').innerHTML = '-- <span class="unit">ms</span>';
|
||||
document.getElementById('checks-body').innerHTML = '<tr><td colspan="6" class="text-muted text-center">No data</td></tr>';
|
||||
return;
|
||||
}
|
||||
|
||||
const latest = data[data.length - 1];
|
||||
|
||||
const statusEl = document.getElementById('status');
|
||||
statusEl.innerHTML = latest.is_up ? '<span class="status-up">✅ UP</span>' : '<span class="status-down">❌ DOWN</span>';
|
||||
|
||||
document.getElementById('last-latency').innerHTML = `${latest.latency_ms || '—'} <span class="unit">ms</span>`;
|
||||
|
||||
const sslDays = latest.ssl_days_remaining;
|
||||
document.getElementById('ssl-status').innerHTML = getSslBadge(sslDays);
|
||||
document.getElementById('ssl-days').textContent = sslDays !== null ? `${sslDays} days remaining` : 'Unknown';
|
||||
|
||||
document.getElementById('dns-latency').innerHTML = `${latest.dns_ms || '—'} <span class="unit">ms</span>`;
|
||||
document.getElementById('search-latency').innerHTML = `${latest.search_latency_ms || '—'} <span class="unit">ms</span>`;
|
||||
|
||||
const uptime = computeUptime(data);
|
||||
document.getElementById('uptime').textContent = uptime.toFixed(2) + '%';
|
||||
|
||||
// Table (last 10)
|
||||
const tbody = document.getElementById('checks-body');
|
||||
tbody.innerHTML = '';
|
||||
const last10 = data.slice(-10).reverse();
|
||||
last10.forEach(entry => {
|
||||
const row = document.createElement('tr');
|
||||
const statusBadge = entry.is_up ? '<span class="badge badge-up">UP</span>' : '<span class="badge badge-down">DOWN</span>';
|
||||
const sslBadge = getSslBadge(entry.ssl_days_remaining);
|
||||
row.innerHTML = `
|
||||
<td>${new Date(entry.timestamp).toLocaleString()}</td>
|
||||
<td>${statusBadge}</td>
|
||||
<td>${entry.latency_ms || '—'}</td>
|
||||
<td>${entry.dns_ms || '—'}</td>
|
||||
<td>${sslBadge}</td>
|
||||
<td>${entry.search_latency_ms || '—'}</td>
|
||||
`;
|
||||
tbody.appendChild(row);
|
||||
});
|
||||
|
||||
document.getElementById('last-update').textContent = `📡 Last updated: ${new Date().toLocaleString()}`;
|
||||
|
||||
// Chart (last 60)
|
||||
const chartData = data.slice(-60);
|
||||
const labels = chartData.map(e => new Date(e.timestamp).toLocaleTimeString());
|
||||
const latencies = chartData.map(e => e.latency_ms || 0);
|
||||
const statuses = chartData.map(e => e.is_up ? 1 : 0);
|
||||
|
||||
if (chart) {
|
||||
chart.data.labels = labels;
|
||||
chart.data.datasets[0].data = latencies;
|
||||
chart.data.datasets[0].pointBackgroundColor = statuses.map(s => s === 1 ? '#28a745' : '#dc3545');
|
||||
chart.update();
|
||||
} else {
|
||||
const ctx = document.getElementById('latencyChart').getContext('2d');
|
||||
chart = new Chart(ctx, {
|
||||
type: 'line',
|
||||
data: {
|
||||
labels: labels,
|
||||
datasets: [{
|
||||
label: 'Latency (ms)',
|
||||
data: latencies,
|
||||
borderColor: '#667eea',
|
||||
backgroundColor: 'rgba(102, 126, 234, 0.1)',
|
||||
tension: 0.2,
|
||||
fill: true,
|
||||
pointRadius: 3,
|
||||
pointBackgroundColor: statuses.map(s => s === 1 ? '#28a745' : '#dc3545'),
|
||||
pointBorderColor: '#ffffff',
|
||||
pointBorderWidth: 1
|
||||
}]
|
||||
},
|
||||
options: {
|
||||
responsive: true,
|
||||
maintainAspectRatio: false,
|
||||
plugins: { legend: { display: false } },
|
||||
scales: {
|
||||
x: { grid: { display: false }, ticks: { maxTicksLimit: 15 } },
|
||||
y: { beginAtZero: true, grid: { color: '#f0f0f0' }, title: { display: true, text: 'Latency (ms)' } }
|
||||
}
|
||||
}
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
async function fetchData() {
|
||||
try {
|
||||
const response = await fetch('monitoring-data.json');
|
||||
if (!response.ok) throw new Error('Network error');
|
||||
const data = await response.json();
|
||||
return Array.isArray(data) ? data : [];
|
||||
} catch (error) {
|
||||
console.error('Failed to fetch data:', error);
|
||||
return [];
|
||||
}
|
||||
}
|
||||
|
||||
async function refresh() {
|
||||
const data = await fetchData();
|
||||
updateDashboard(data);
|
||||
}
|
||||
|
||||
refresh();
|
||||
setInterval(refresh, 10000);
|
||||
</script>
|
||||
</body>
|
||||
</html>
|
||||
@@ -0,0 +1,174 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
Monitoring Script for mithal.space
|
||||
Collects: latency, uptime, SSL, DNS, search response
|
||||
"""
|
||||
|
||||
import requests
|
||||
import json
|
||||
import time
|
||||
import socket
|
||||
import ssl
|
||||
import datetime
|
||||
import os
|
||||
from urllib.parse import urlparse
|
||||
|
||||
# Configuration
|
||||
TARGET_URL = "https://mithal.space"
|
||||
SEARCH_QUERY = "?q=test" # adjust if search endpoint differs
|
||||
DATA_FILE = "monitoring-data.json"
|
||||
MAX_ENTRIES = 1440 # 24 hours * 60 minutes
|
||||
TIMEOUT = 10
|
||||
|
||||
def check_ssl_certificate(hostname, port=443):
|
||||
"""Check SSL certificate expiry date."""
|
||||
try:
|
||||
context = ssl.create_default_context()
|
||||
with socket.create_connection((hostname, port), timeout=TIMEOUT) as sock:
|
||||
with context.wrap_socket(sock, server_hostname=hostname) as ssock:
|
||||
cert = ssock.getpeercert()
|
||||
expiry_date = datetime.datetime.strptime(cert['notAfter'], '%b %d %H:%M:%S %Y %Z')
|
||||
days_remaining = (expiry_date - datetime.datetime.utcnow()).days
|
||||
return {
|
||||
"expiry_date": expiry_date.isoformat(),
|
||||
"days_remaining": days_remaining,
|
||||
"valid": days_remaining > 0
|
||||
}
|
||||
except Exception as e:
|
||||
return {
|
||||
"expiry_date": None,
|
||||
"days_remaining": None,
|
||||
"valid": False,
|
||||
"error": str(e)
|
||||
}
|
||||
|
||||
def resolve_dns(hostname):
|
||||
"""Measure DNS resolution time."""
|
||||
try:
|
||||
start = time.time()
|
||||
socket.gethostbyname(hostname)
|
||||
dns_time = round((time.time() - start) * 1000, 2)
|
||||
return dns_time
|
||||
except Exception:
|
||||
return None
|
||||
|
||||
def check_latency_and_uptime(url):
|
||||
"""Check HTTP response time and status code."""
|
||||
try:
|
||||
start = time.time()
|
||||
response = requests.get(url, timeout=TIMEOUT)
|
||||
latency = round((time.time() - start) * 1000, 2)
|
||||
status_code = response.status_code
|
||||
is_up = 200 <= status_code < 400
|
||||
return {
|
||||
"latency": latency,
|
||||
"status_code": status_code,
|
||||
"is_up": is_up,
|
||||
"error": None
|
||||
}
|
||||
except requests.RequestException as e:
|
||||
return {
|
||||
"latency": None,
|
||||
"status_code": None,
|
||||
"is_up": False,
|
||||
"error": str(e)
|
||||
}
|
||||
|
||||
def check_search_response(url, query="?q=test"):
|
||||
"""Measure search endpoint response time."""
|
||||
try:
|
||||
search_url = f"{url}{query}"
|
||||
start = time.time()
|
||||
response = requests.get(search_url, timeout=TIMEOUT)
|
||||
search_latency = round((time.time() - start) * 1000, 2)
|
||||
return {
|
||||
"search_latency": search_latency,
|
||||
"search_status": response.status_code,
|
||||
"search_error": None
|
||||
}
|
||||
except requests.RequestException as e:
|
||||
return {
|
||||
"search_latency": None,
|
||||
"search_status": None,
|
||||
"search_error": str(e)
|
||||
}
|
||||
|
||||
def collect_metrics():
|
||||
"""Collect all metrics for mithal.space."""
|
||||
parsed = urlparse(TARGET_URL)
|
||||
hostname = parsed.hostname or "mithal.space"
|
||||
|
||||
dns_time = resolve_dns(hostname)
|
||||
ssl_info = check_ssl_certificate(hostname)
|
||||
health = check_latency_and_uptime(TARGET_URL)
|
||||
search = check_search_response(TARGET_URL, SEARCH_QUERY)
|
||||
|
||||
metrics = {
|
||||
"timestamp": datetime.datetime.utcnow().isoformat() + "Z",
|
||||
"url": TARGET_URL,
|
||||
"dns_ms": dns_time,
|
||||
"ssl_valid": ssl_info.get("valid", False),
|
||||
"ssl_days_remaining": ssl_info.get("days_remaining"),
|
||||
"ssl_expiry_date": ssl_info.get("expiry_date"),
|
||||
"latency_ms": health.get("latency"),
|
||||
"status_code": health.get("status_code"),
|
||||
"is_up": health.get("is_up", False),
|
||||
"error": health.get("error"),
|
||||
"search_latency_ms": search.get("search_latency"),
|
||||
"search_status_code": search.get("search_status"),
|
||||
"search_error": search.get("search_error")
|
||||
}
|
||||
return metrics
|
||||
|
||||
def save_metrics(metrics, filename=DATA_FILE, max_entries=MAX_ENTRIES):
|
||||
"""Save metrics to JSON file with size limit."""
|
||||
try:
|
||||
if os.path.exists(filename):
|
||||
with open(filename, 'r') as f:
|
||||
data = json.load(f)
|
||||
else:
|
||||
data = []
|
||||
data.append(metrics)
|
||||
if len(data) > max_entries:
|
||||
data = data[-max_entries:]
|
||||
with open(filename, 'w') as f:
|
||||
json.dump(data, f, indent=2)
|
||||
return True
|
||||
except Exception as e:
|
||||
print(f"Error saving data: {e}")
|
||||
return False
|
||||
|
||||
def run_monitor():
|
||||
"""Main monitoring loop."""
|
||||
print(f"🚀 Starting monitoring for {TARGET_URL}")
|
||||
print(f"📊 Data will be saved to {DATA_FILE}")
|
||||
print("=" * 50)
|
||||
|
||||
while True:
|
||||
try:
|
||||
print(f"[{datetime.datetime.now().strftime('%Y-%m-%d %H:%M:%S')}] Collecting metrics...")
|
||||
metrics = collect_metrics()
|
||||
|
||||
status = "✅ UP" if metrics['is_up'] else "❌ DOWN"
|
||||
print(f" Status: {status}")
|
||||
print(f" Latency: {metrics['latency_ms']} ms")
|
||||
print(f" DNS: {metrics['dns_ms']} ms")
|
||||
print(f" SSL: {metrics['ssl_days_remaining']} days remaining")
|
||||
print(f" Search: {metrics['search_latency_ms']} ms")
|
||||
|
||||
if save_metrics(metrics):
|
||||
print(" ✅ Data saved")
|
||||
else:
|
||||
print(" ❌ Failed to save data")
|
||||
print("-" * 50)
|
||||
|
||||
except KeyboardInterrupt:
|
||||
print("\n🛑 Monitoring stopped by user")
|
||||
break
|
||||
except Exception as e:
|
||||
print(f"❌ Error in monitoring loop: {e}")
|
||||
|
||||
time.sleep(60)
|
||||
|
||||
if __name__ == "__main__":
|
||||
run_monitor()
|
||||
1
q5-mithal-monitor/requirements.txt
Normal file
1
q5-mithal-monitor/requirements.txt
Normal file
@@ -0,0 +1 @@
|
||||
requests==2.31.0
|
||||
المرجع في مشكلة جديدة
حظر مستخدم