Complete Ghaymah Internship Assessment

README.md

@@ -0,0 +1,126 @@
+# 🚀 Ghaymah SRE Internship Assessment
+
+## 📌 Overview
+
+This repository contains my complete solutions for the **Ghaymah Site Reliability Engineering (SRE) Internship Assessment**.
+
+The assessment focuses on real-world SRE practices, including **Service Monitoring**, **Incident Management**, **Reliability Engineering**, **CI/CD**, **Scalability**, and **Production Operations**.
+
+Each task has been completed with an emphasis on reliability, automation, observability, and production-ready engineering practices.
+
+---
+
+## 📂 Project Structure
+
+```text
+.
+├── q1-deploy-monitor/
+├── q2-postmortem/
+├── q3-cicd/
+├── q4-scalability/
+├── q5-mithal-monitor/
+├── common-mortakaz/
+├── common-qabilah/
+└── README.md
+```
+
+---
+
+## 📋 Assessment Tasks
+
+### ✅ Q1 – Deploy & Monitor
+
+* Dockerized API Deployment
+* Health Check Endpoint
+* Monitoring Script
+* Monitoring Dashboard
+
+### ✅ Q2 – Incident Postmortem
+
+* Incident Summary
+* Timeline Analysis
+* Root Cause Analysis
+* Corrective & Preventive Actions
+* Auto Scaling Policy
+* Early Detection Strategy
+
+### ✅ Q3 – CI/CD Pipeline
+
+* GitHub Actions Workflow
+* Docker Image Build
+* Container Registry Push
+* Manual Approval for Production
+* Staging vs Production
+* Ghaymah CLI Integration
+
+### ✅ Q4 – Scalability & Architecture
+
+* High-Level Architecture Design
+* Capacity Planning
+* Container Scaling Calculations
+* Cold Start Strategy
+* Block Storage for Stateful Applications
+
+### ✅ Q5 – Monitoring Dashboard
+
+Monitoring **mithal.space** by collecting and visualizing:
+
+* HTTP Latency
+* Uptime Status
+* SSL Certificate Health
+* DNS Resolution Time
+* Search Response Time
+* Historical Metrics
+* Interactive Monitoring Dashboard
+
+---
+
+## 🛠 Technologies Used
+
+* Docker
+* Python
+* Bash
+* HTML5
+* CSS3
+* JavaScript
+* GitHub Actions
+* Markdown
+
+---
+
+## 🎯 SRE Concepts Demonstrated
+
+* Service Reliability
+* Health Checks
+* Monitoring & Observability
+* Incident Response
+* Postmortem Analysis
+* Capacity Planning
+* Horizontal Scaling
+* Production Readiness
+* CI/CD Automation
+* Infrastructure Monitoring
+
+---
+
+## 👨‍💻 About Me
+
+**Sayed Atwa**
+
+**Site Reliability Engineering (SRE) | Linux | Cloud | Automation**
+
+📧 **Email:** [sayed.atwh.sayed@gmail.com](mailto:sayed.atwh.sayed@gmail.com)
+
+📱 **Phone:** +20 110 155 8236
+
+🔗 **LinkedIn:** https://www.linkedin.com/in/sayed-atwh-sayed
+
+---
+
+## 📖 Purpose
+
+This repository represents my practical implementation of the Ghaymah SRE Internship challenges and demonstrates my ability to design reliable systems, automate operational workflows, analyze incidents, and build monitoring solutions following modern Site Reliability Engineering practices.
+
+---
+
+⭐ **Thank you for reviewing my submission. I appreciate the opportunity to participate in the Ghaymah SRE Internship Assessment.**

cicd/workflow.yml

@@ -0,0 +1,97 @@
+name: Build and Deploy
+
+on:
+  push:
+    branches:
+      - main
+
+  workflow_dispatch:
+
+jobs:
+  build:
+    runs-on: ubuntu-latest
+
+    steps:
+      - name: Checkout Repository
+        uses: actions/checkout@v4
+
+      - name: Set up Docker Buildx
+        uses: docker/setup-buildx-action@v3
+
+      - name: Login to Ghaymah Container Registry
+        run: |
+          echo "${{ secrets.GHAYMAH_TOKEN }}" | docker login registry.ghaymah.systems \
+          --username ${{ secrets.GHAYMAH_USERNAME }} \
+          --password-stdin
+
+      - name: Build Docker Image
+        run: |
+          docker build -t registry.ghaymah.systems/sample-api:${{ github.sha }} .
+
+      - name: Push Docker Image
+        run: |
+          docker push registry.ghaymah.systems/sample-api:${{ github.sha }}
+
+  deploy-staging:
+    needs: build
+    runs-on: ubuntu-latest
+
+    steps:
+      - name: Deploy to Staging
+        run: |
+          ghaymah deploy \
+          --environment staging \
+          --image registry.ghaymah.systems/sample-api:${{ github.sha }}
+
+  deploy-production:
+    needs: deploy-staging
+    runs-on: ubuntu-latest
+
+    environment:
+      name: production
+
+    steps:
+      - name: Manual Approval
+        run: echo "Waiting for manual approval..."
+
+      - name: Deploy to Production
+        run: |
+          ghaymah deploy \
+          --environment production \
+          --image registry.ghaymah.systems/sample-api:${{ github.sha }}
+
+# environment:
+#  name: production
+
+################ Staging و Production
+
+# Staging:
+# - Testing environment
+# - Used before production
+# - Safe for validation
+
+# Production:
+# - Live environment
+# - Real users
+# - Requires manual approval
+# - High availability and monitoring
+
+
+###########################  Ghaymah CLI
+
+# Ghaymah CLI Documentation
+#
+# Install CLI
+# curl -fsSL https://ghaymah.systems/install.sh | bash
+#
+# Login
+# ghaymah login
+#
+# Verify
+# ghaymah whoami
+#
+# Deploy
+# ghaymah deploy --environment staging
+#
+# Deploy Production
+# ghaymah deploy --environment production

common-mortakaz/integration-1.md

@@ -0,0 +1,17 @@
+# Mortakaz Integration 1
+
+## Objective
+
+Integrate an application with Mortakaz services.
+
+## Integration Steps
+
+1. Authenticate with the Mortakaz platform.
+2. Configure API credentials.
+3. Test API connectivity.
+4. Validate request and response data.
+5. Monitor integration logs.
+
+## Expected Result
+
+The application communicates successfully with Mortakaz services and returns valid responses.

common-mortakaz/integration-2.md

@@ -0,0 +1,23 @@
+# Mortakaz Integration 2
+
+## Monitoring
+
+- Enable logging.
+- Enable health checks.
+- Configure alerts.
+- Monitor API latency.
+- Monitor error rate.
+
+## Security
+
+- HTTPS only
+- API Token Authentication
+- Rate Limiting
+- Secure Secret Management
+
+## Best Practices
+
+- Retry failed requests.
+- Handle timeouts gracefully.
+- Validate API responses.
+- Monitor service availability.

common-qabilah/qabilah-profile.txt

@@ -0,0 +1,15 @@
+Qabilah Platform Profile
+
+Platform: Qabilah
+
+Purpose:
+Community collaboration and digital platform integration.
+
+Features:
+- Authentication
+- API Integration
+- Monitoring
+- Logging
+- Deployment Support
+- Scalability
+- Security Best Practices

deploy-monitor/Dockerfile

@@ -0,0 +1,24 @@
+FROM python:3.11-slim
+
+WORKDIR /app
+
+RUN printf 'from http.server import BaseHTTPRequestHandler, HTTPServer\n\
+import json\n\
+class Handler(BaseHTTPRequestHandler):\n\
+    requests = 0\n\
+    def do_GET(self):\n\
+        Handler.requests += 1\n\
+        if self.path == "/health":\n\
+            self.send_response(200)\n\
+            self.send_header("Content-Type","application/json")\n\
+            self.end_headers()\n\
+            self.wfile.write(json.dumps({"status":"UP","requests":Handler.requests}).encode())\n\
+        else:\n\
+            self.send_response(200)\n\
+            self.end_headers()\n\
+            self.wfile.write(b"Hello from Ghaymah!")\n\
+HTTPServer(("0.0.0.0",8000),Handler).serve_forever()' > app.py
+
+EXPOSE 8000
+
+CMD ["python","app.py"]

deploy-monitor/dashboard.html

@@ -0,0 +1,108 @@
+<!DOCTYPE html>
+<html lang="en">
+<head>
+<meta charset="UTF-8">
+<title>API Dashboard</title>
+
+<style>
+
+body{
+    font-family:Arial,sans-serif;
+    background:#f5f5f5;
+    text-align:center;
+    margin-top:80px;
+}
+
+.card{
+    width:350px;
+    margin:auto;
+    padding:30px;
+    background:white;
+    border-radius:10px;
+    box-shadow:0 0 10px rgba(0,0,0,.2);
+}
+
+h2{
+    color:#333;
+}
+
+.status{
+    font-size:25px;
+    margin:20px;
+    font-weight:bold;
+}
+
+.response{
+    font-size:20px;
+    margin:15px;
+}
+
+.requests{
+    font-size:20px;
+    margin:15px;
+}
+
+</style>
+
+</head>
+
+<body>
+
+<div class="card">
+
+<h2>API Monitoring Dashboard</h2>
+
+<div class="status" id="status">Checking...</div>
+
+<div class="response" id="response"></div>
+
+<div class="requests" id="requests"></div>
+
+</div>
+
+<script>
+
+async function checkHealth(){
+
+let start=performance.now();
+
+try{
+
+const res=await fetch("http://localhost:8000/health");
+
+const data=await res.json();
+
+let end=performance.now();
+
+document.getElementById("status").innerHTML="🟢 UP";
+
+document.getElementById("status").style.color="green";
+
+document.getElementById("response").innerHTML="Response Time : "+(end-start).toFixed(2)+" ms";
+
+document.getElementById("requests").innerHTML="Requests : "+data.requests;
+
+}
+
+catch{
+
+document.getElementById("status").innerHTML="🔴 DOWN";
+
+document.getElementById("status").style.color="red";
+
+document.getElementById("response").innerHTML="-";
+
+document.getElementById("requests").innerHTML="-";
+
+}
+
+}
+
+checkHealth();
+
+setInterval(checkHealth,30000);
+
+</script>
+
+</body>
+</html>

deploy-monitor/health-check.sh

@@ -0,0 +1,20 @@
+#!/bin/bash
+
+URL="http://localhost:8000/health"
+
+echo "Starting Health Monitor..."
+
+while true
+do
+    RESPONSE=$(curl -s -o /dev/null -w "%{http_code}" $URL)
+    TIME=$(curl -s -o /dev/null -w "%{time_total}" $URL)
+    DATE=$(date +"%Y-%m-%d %H:%M:%S")
+
+    if [ "$RESPONSE" = "200" ]; then
+        echo "[$DATE] STATUS: UP | HTTP: $RESPONSE | Response Time: ${TIME}s"
+    else
+        echo "[$DATE] STATUS: DOWN | HTTP: $RESPONSE"
+    fi
+
+    sleep 30
+done

mithal-monitor/dashboard.html

@@ -0,0 +1,174 @@
+<!DOCTYPE html>
+<html>
+<head>
+
+<meta charset="UTF-8">
+
+<title>Mithal Monitoring Dashboard</title>
+
+<script src="https://cdn.jsdelivr.net/npm/chart.js"></script>
+
+<style>
+
+body{
+font-family:Arial;
+background:#f5f5f5;
+padding:30px;
+}
+
+.card{
+background:white;
+padding:20px;
+margin:20px;
+border-radius:10px;
+box-shadow:0 0 10px #ccc;
+}
+
+table{
+width:100%;
+border-collapse:collapse;
+}
+
+th,td{
+padding:10px;
+border:1px solid #ddd;
+text-align:center;
+}
+
+</style>
+
+</head>
+
+<body>
+
+<h1>Mithal.space Monitoring Dashboard</h1>
+
+<div class="card">
+
+<h2>Uptime (24h)</h2>
+
+<div id="uptime"></div>
+
+</div>
+
+<div class="card">
+
+<h2>SSL Status</h2>
+
+<div id="ssl"></div>
+
+</div>
+
+<div class="card">
+
+<h2>Latency (Last Hour)</h2>
+
+<canvas id="chart"></canvas>
+
+</div>
+
+<div class="card">
+
+<h2>Last 10 Checks</h2>
+
+<table>
+
+<thead>
+
+<tr>
+
+<th>Time</th>
+
+<th>Status</th>
+
+<th>Latency</th>
+
+<th>DNS</th>
+
+<th>Search</th>
+
+</tr>
+
+</thead>
+
+<tbody id="table"></tbody>
+
+</table>
+
+</div>
+
+<script>
+
+fetch("monitor_data.json")
+
+.then(r=>r.json())
+
+.then(data=>{
+
+let success=data.filter(x=>x.uptime).length;
+
+let uptime=((success/data.length)*100).toFixed(2);
+
+document.getElementById("uptime").innerHTML=uptime+" %";
+
+let last=data[data.length-1];
+
+document.getElementById("ssl").innerHTML=
+
+last.ssl_remaining_days+" days remaining";
+
+let hour=data.slice(-60);
+
+new Chart(document.getElementById("chart"),{
+
+type:"line",
+
+data:{
+
+labels:hour.map(x=>x.time),
+
+datasets:[{
+
+label:"Latency",
+
+data:hour.map(x=>x.latency)
+
+}]
+
+}
+
+});
+
+let html="";
+
+data.slice(-10).reverse().forEach(x=>{
+
+html+=`
+
+<tr>
+
+<td>${x.time}</td>
+
+<td>${x.status}</td>
+
+<td>${x.latency} ms</td>
+
+<td>${x.dns} ms</td>
+
+<td>${x.search_response} ms</td>
+
+</tr>
+
+`;
+
+});
+
+document.getElementById("table").innerHTML=html;
+
+});
+
+</script>
+
+</body>
+
+</html>

mithal-monitor/monitor.py

@@ -0,0 +1,103 @@
+import requests
+import socket
+import ssl
+import time
+import json
+import os
+from datetime import datetime
+
+URL = "https://mithal.space"
+SEARCH_URL = "https://mithal.space/search?q=test"
+
+DATA_FILE = "monitor_data.json"
+
+
+def get_latency():
+    start = time.time()
+    r = requests.get(URL, timeout=10)
+    latency = (time.time() - start) * 1000
+    return round(latency, 2), r.status_code
+
+
+def get_dns_time():
+    start = time.time()
+    socket.gethostbyname("mithal.space")
+    dns = (time.time() - start) * 1000
+    return round(dns, 2)
+
+
+def get_ssl():
+    hostname = "mithal.space"
+
+    ctx = ssl.create_default_context()
+
+    with ctx.wrap_socket(socket.socket(), server_hostname=hostname) as s:
+        s.settimeout(5)
+        s.connect((hostname, 443))
+        cert = s.getpeercert()
+
+    expiry = cert["notAfter"]
+
+    expire_date = datetime.strptime(expiry, "%b %d %H:%M:%S %Y %Z")
+
+    remaining = (expire_date - datetime.utcnow()).days
+
+    return expiry, remaining
+
+
+def get_search_time():
+
+    start = time.time()
+
+    try:
+        requests.get(SEARCH_URL, timeout=10)
+    except:
+        pass
+
+    return round((time.time() - start) * 1000, 2)
+
+
+def save():
+
+    latency, status = get_latency()
+
+    dns = get_dns_time()
+
+    ssl_expiry, ssl_days = get_ssl()
+
+    search = get_search_time()
+
+    data = {
+        "time": datetime.now().strftime("%Y-%m-%d %H:%M:%S"),
+        "latency": latency,
+        "status": status,
+        "uptime": status == 200,
+        "dns": dns,
+        "ssl_expiry": ssl_expiry,
+        "ssl_remaining_days": ssl_days,
+        "search_response": search
+    }
+
+    if os.path.exists(DATA_FILE):
+        with open(DATA_FILE, "r") as f:
+            history = json.load(f)
+    else:
+        history = []
+
+    history.append(data)
+
+    history = history[-1440:]
+
+    with open(DATA_FILE, "w") as f:
+        json.dump(history, f, indent=4)
+
+
+if __name__ == "__main__":
+
+    while True:
+
+        save()
+
+        print("Saved:", datetime.now())
+
+        time.sleep(60)

postmortem/postmortem-report.md

@@ -0,0 +1,124 @@
+# Postmortem Report
+
+## Incident Summary
+
+**Incident:** Application outage due to repeated OOMKilled events.
+
+**Duration:** 45 minutes
+
+**Impact:**
+- The application was unavailable for users.
+- API requests failed during the outage.
+- User experience was significantly affected.
+
+---
+
+## Timeline
+
+| Time | Event |
+|------|-------|
+| 10:00 | Application deployed successfully. |
+| 10:08 | Memory usage started increasing rapidly. |
+| 10:12 | First OOMKilled event occurred. |
+| 10:13 | Kubernetes restarted the container. |
+| 10:15–10:40 | Continuous OOMKilled restart loop. |
+| 10:42 | Engineering team investigated the issue. |
+| 10:45 | Memory limit increased and application stabilized. |
+
+---
+
+## Root Cause
+
+The application exceeded its allocated memory limit.
+
+The container was repeatedly terminated by Kubernetes with an **OOMKilled** event because memory consumption continued growing beyond the configured limit.
+
+Possible contributing factors:
+
+- Memory leak inside the application.
+- Memory limits configured too low.
+- No Horizontal Pod Autoscaler.
+- Lack of memory usage alerts.
+
+---
+
+## Resolution
+
+The engineering team:
+
+- Increased container memory limits.
+- Restarted the deployment.
+- Verified application health.
+- Monitored memory consumption until stable.
+
+---
+
+## Recommendations
+
+### Immediate Actions
+
+- Configure appropriate memory requests and limits.
+- Investigate memory leaks.
+- Enable memory monitoring.
+- Configure alerting.
+
+### Long-Term Improvements
+
+- Implement Horizontal Pod Autoscaler (HPA).
+- Perform load testing before production.
+- Review memory usage after each deployment.
+- Add automatic scaling policies.
+- Conduct regular post-deployment monitoring.
+
+---
+
+# Auto-Scaling Policy
+
+To prevent similar incidents, the platform should implement:
+
+- Minimum replicas: **2**
+- Maximum replicas: **10**
+- Scale out when:
+  - Memory usage > 70%
+  - CPU usage > 70%
+- Scale in when:
+  - Memory usage < 40%
+  - CPU usage < 40%
+- Cooldown period: **5 minutes**
+- Enable automatic replacement of unhealthy pods.
+
+This policy ensures enough capacity during traffic spikes while reducing resource waste during normal operation.
+
+---
+
+# Early Detection Using Ghaymah Monitoring
+
+The issue can be detected early by monitoring:
+
+- Memory usage
+- CPU usage
+- Container restart count
+- OOMKilled events
+- Pod health status
+- Response time
+- Error rate
+
+Recommended alerts:
+
+- Memory usage above 80%
+- More than 3 container restarts within 10 minutes
+- Pod enters CrashLoopBackOff
+- Health endpoint becomes unavailable
+- Response time exceeds acceptable thresholds
+
+These monitoring practices allow engineers to respond before users experience service interruption.
+
+---
+
+# Lessons Learned
+
+- Proper resource limits are essential.
+- Continuous monitoring is critical.
+- Autoscaling improves application availability.
+- Early alerting reduces downtime.
+- Capacity planning should be reviewed before production deployments.

scalability/calculations.md

@@ -0,0 +1,68 @@
+# Scalability Calculations
+
+## Required Capacity
+
+Expected traffic:
+
+15,000 requests/second
+
+Each container handles:
+
+500 requests/second
+
+Base number of containers:
+
+15000 / 500 = 30 containers
+
+---
+
+## Safety Margin
+
+Required safety margin:
+
+30%
+
+30 × 1.30 = 39 containers
+
+Final requirement:
+
+**39 containers**
+
+---
+
+# Cold Start Strategy
+
+To reduce startup latency for new containers:
+
+- Keep at least 2 warm standby containers.
+- Pre-pull container images on worker nodes.
+- Use lightweight container images.
+- Enable Horizontal Pod Autoscaler.
+- Scale gradually based on CPU and request rate.
+- Perform health checks before routing traffic.
+
+This minimizes cold start delays during traffic spikes.
+
+---
+
+# Using Ghaymah Block Storage
+
+Block Storage is suitable for stateful workloads because it provides persistent storage independent of the container lifecycle.
+
+Typical use cases:
+
+- PostgreSQL
+- MySQL
+- MongoDB
+- Redis persistence
+- Application uploads
+- Log storage
+
+Benefits:
+
+- Persistent data after container restarts.
+- High-performance disk access.
+- Easy attachment to containers.
+- Reliable storage for production workloads.
+
+Containers can be recreated without losing application data because the storage volume remains intact.