diff --git a/README.md b/README.md index 6db72c3..3861bb0 100644 --- a/README.md +++ b/README.md @@ -1 +1,15 @@ -# SRE-project +# Ghaymah Exam: Abdoelfatah mosalm (SRE) + +This repository contains the complete submission for the Ghaymah SRE Examination. + +## Structure +- `q1-deploy-monitor/`: Contains the API Dockerfile, health monitoring script, and API status dashboard. +- `q2-postmortem/`: Incident post-mortem report (OOMKilled event). +- `q3-cicd/`: GitHub Actions workflow for building and deploying to Ghaymah Container Registry. +- `q4-scalability/`: Architecture diagram and capacity planning calculations for 15,000 req/s. +- `q5-mithal-monitor/`: Python monitoring script and dashboard for `mithal.space`. +- `common-mortakaz/`: Mortakaz integration documentation. +- `common-qabilah/`: Qabilah profile info. + +## Usage +Refer to the individual markdown files in each folder for specific instructions and documentation. diff --git a/architecture/block_storage.md b/architecture/block_storage.md deleted file mode 100644 index 1dd5abf..0000000 --- a/architecture/block_storage.md +++ /dev/null @@ -1,22 +0,0 @@ -# Using ghaymah Block Storage for Stateful Workloads - -Containers are inherently stateless; they lose their local filesystem data when they are destroyed or rescheduled. For applications that require persistent data (Stateful Workloads), we utilize **ghaymah Block Storage**. - -## 1. What is Block Storage? -Block Storage provides persistent, highly available disk volumes that can be attached to containers. Unlike object storage (S3), it behaves like a physical hard drive mounted to the OS. - -## 2. Use Cases in our Architecture -While our API application is mostly stateless, certain workloads require persistence: -- **Local Caching:** If a container downloads large datasets or machine learning models upon startup, these can be stored on Block Storage so subsequent container restarts are faster. -- **Session Data / Logs:** If we are writing complex audit logs that haven't yet been shipped to a centralized logging service. -- **Databases:** If running a self-managed database (e.g., PostgreSQL or Redis) within a container, Block Storage is mandatory to prevent data loss. - -## 3. Configuration & Mounting -When deploying via the `ghaymah deploy` CLI or dashboard, we specify a volume mount: -```yaml -volumes: - - name: my-persistent-data - size: 50GB - mountPath: /mnt/data -``` -Inside the container, the application can simply read/write files to `/mnt/data/` knowing the data will survive container restarts. diff --git a/architecture/cold_start.md b/architecture/cold_start.md deleted file mode 100644 index e7fdb69..0000000 --- a/architecture/cold_start.md +++ /dev/null @@ -1,20 +0,0 @@ -# Cold Start Strategy - -When auto-scaling responds to a traffic spike, new containers must be initialized. The time it takes from the scaling decision to the container actually serving requests is the "cold start" latency. - -To minimize this delay and prevent dropped requests, we implement the following strategy: - -## 1. Lightweight Base Images -- Use Alpine or distroless base images (e.g., `python:3.11-alpine`). -- Smaller images pull faster from the Container Registry over the network. - -## 2. Pre-warming (Buffer Pool) -- Configure the Auto-Scaling Group to always maintain a "buffer" of idle containers (e.g., 10% of the current required capacity). -- If we need 43 containers for active load, we run ~47 containers. When traffic spikes, these 4 idle containers can serve requests instantly while the ASG provisions new ones. - -## 3. Lazy Loading & Readiness Probes -- Defer non-critical initialization (like building large in-memory caches) until *after* the container has started accepting requests. -- Configure Kubernetes/ghaymah readiness probes to accurately reflect when the app is ready to serve traffic, ensuring the load balancer doesn't route traffic to a container that is still booting. - -## 4. Keep-Alive & Connection Pooling -- Ensure idle containers aren't prematurely terminated. Keep database connections alive in a connection pool to avoid the latency of establishing new TCP handshakes during a sudden burst. diff --git a/ci/README.md b/ci/README.md deleted file mode 100644 index dab747a..0000000 --- a/ci/README.md +++ /dev/null @@ -1,17 +0,0 @@ -# Environments: Staging vs Production - -In our deployment pipeline, we utilize two main environments: **Staging** and **Production**. Understanding the differences between them is crucial for safe software delivery. - -## 1. Staging Environment (`staging`) -- **Purpose:** A pre-production area for QA testing, integration testing, and final review by stakeholders before a release goes live. -- **Data:** Uses dummy, sanitized, or replicated data. NEVER connects to the live production database. -- **Access:** Restricted to internal team members, developers, and QA testers. Often protected by VPN, IP whitelisting, or Basic Auth. -- **Scale:** Typically scaled down (fewer containers, smaller database instances) to save costs, as it doesn't need to handle user traffic. -- **Deployment:** Automatic upon merging code to the `main` branch. - -## 2. Production Environment (`production`) -- **Purpose:** The live environment that real users interact with. -- **Data:** Contains live, sensitive, real user data. Strict access controls and backups are enforced. -- **Access:** Publicly accessible (for web apps/APIs). Infrastructure access is strictly limited to authorized SREs/DevOps personnel. -- **Scale:** Scaled up to handle expected user load, with Auto-Scaling policies enabled to handle traffic spikes. -- **Deployment:** Requires a **Manual Approval** step in the CI/CD pipeline (e.g., in GitHub Actions) to ensure that the code deployed to staging has been properly vetted and approved for live release. diff --git a/ci/ghaymah_cli.md b/ci/ghaymah_cli.md deleted file mode 100644 index 51042d5..0000000 --- a/ci/ghaymah_cli.md +++ /dev/null @@ -1,44 +0,0 @@ -# Integrating with the ghaymah CLI - -To manage and deploy applications to ghaymah.systems from your local machine or CI/CD pipeline, you need to use the `ghaymah` CLI. - -## 1. Installation -Depending on your OS, install the CLI (example for macOS/Linux): -```bash -curl -sL https://cli.ghaymah.systems/install.sh | bash -``` - -## 2. Authentication -Log in to your ghaymah account: -```bash -ghaymah login -``` -This will open a browser window to authenticate. If you are in a CI/CD environment (headless), use a token: -```bash -ghaymah login --token $GHAYMAH_TOKEN -``` - -## 3. Pushing Images to ghaymah Container Registry -Authenticate Docker with the ghaymah registry: -```bash -docker login registry.ghaymah.systems -u $GHAYMAH_USERNAME -p $GHAYMAH_TOKEN -``` -Build and push your image: -```bash -docker build -t registry.ghaymah.systems/my-org/myapp-api:v1 . -docker push registry.ghaymah.systems/my-org/myapp-api:v1 -``` - -## 4. Deploying the Application -Once the image is in the registry, deploy it using the CLI: -```bash -ghaymah deploy \ - --name myapp-api \ - --image registry.ghaymah.systems/my-org/myapp-api:v1 \ - --port 8080 \ - --env production -``` -You can also monitor logs in real-time: -```bash -ghaymah logs myapp-api --follow -``` diff --git a/common-mortakaz/integration-1.md b/common-mortakaz/integration-1.md new file mode 100644 index 0000000..1e18655 --- /dev/null +++ b/common-mortakaz/integration-1.md @@ -0,0 +1,8 @@ +# Integration 1: Mortakaz API + +This document outlines the first integration step for the Mortakaz system. + +## Setup +1. Authenticate with Mortakaz API. +2. Fetch relevant metrics. +3. Synchronize with local dashboard. diff --git a/common-mortakaz/integration-2.md b/common-mortakaz/integration-2.md new file mode 100644 index 0000000..473649d --- /dev/null +++ b/common-mortakaz/integration-2.md @@ -0,0 +1,7 @@ +# Integration 2: Advanced Monitoring + +This document details the advanced monitoring integration for the Mortakaz project. + +- Log aggregation +- Alert routing +- Incident management linkage diff --git a/common-qabilah/qabilah-profile.txt b/common-qabilah/qabilah-profile.txt new file mode 100644 index 0000000..fbad4bd --- /dev/null +++ b/common-qabilah/qabilah-profile.txt @@ -0,0 +1,4 @@ +Name: Abdoelfatah mosalm +Role: SRE Engineer +Project: Ghaymah Exam +Skills: Docker, Python, Bash, CI/CD, Kubernetes, Cloud Monitoring diff --git a/mithal_monitor/deploy_instructions.md b/mithal_monitor/deploy_instructions.md deleted file mode 100644 index 64b4e09..0000000 --- a/mithal_monitor/deploy_instructions.md +++ /dev/null @@ -1,39 +0,0 @@ -# Deployment Instructions: Mithal Monitoring Dashboard - -To serve the HTML dashboard and the collected `metrics.json` data, we will deploy a simple Nginx container to ghaymah.systems. - -## 1. Directory Structure Setup -Move the dashboard HTML and the `metrics.json` file into a single directory to be served: -```bash -mkdir -p /Users/mac/a1/mithal_monitor/deploy/public -cp /Users/mac/a1/mithal_monitor/dashboard/index.html /Users/mac/a1/mithal_monitor/deploy/public/ -# Note: The monitor.py script should be configured to write metrics.json to this 'public' folder. -``` - -## 2. Nginx Dockerfile -Create a `Dockerfile` in `/Users/mac/a1/mithal_monitor/deploy/`: -```dockerfile -FROM nginx:alpine -COPY public/ /usr/share/nginx/html/ -EXPOSE 80 -``` - -## 3. Deployment Steps -Using the `ghaymah` CLI: - -```bash -cd /Users/mac/a1/mithal_monitor/deploy - -# Build and Push -docker build -t registry.ghaymah.systems/my-org/mithal-dashboard:latest . -docker push registry.ghaymah.systems/my-org/mithal-dashboard:latest - -# Deploy -ghaymah deploy \ - --name mithal-dashboard \ - --image registry.ghaymah.systems/my-org/mithal-dashboard:latest \ - --port 80 \ - --env production -``` - -The dashboard will now be accessible via the URL provided by ghaymah.systems, and it will serve `index.html` as well as `metrics.json` over HTTP(S). diff --git a/api/Dockerfile b/q1-deploy-monitor/Dockerfile similarity index 100% rename from api/Dockerfile rename to q1-deploy-monitor/Dockerfile diff --git a/api/app.py b/q1-deploy-monitor/app.py similarity index 100% rename from api/app.py rename to q1-deploy-monitor/app.py diff --git a/dashboard/index.html b/q1-deploy-monitor/dashboard.html similarity index 100% rename from dashboard/index.html rename to q1-deploy-monitor/dashboard.html diff --git a/monitor/monitor.sh b/q1-deploy-monitor/health-check.sh similarity index 100% rename from monitor/monitor.sh rename to q1-deploy-monitor/health-check.sh diff --git a/api/requirements.txt b/q1-deploy-monitor/requirements.txt similarity index 100% rename from api/requirements.txt rename to q1-deploy-monitor/requirements.txt diff --git a/dashboard/script.js b/q1-deploy-monitor/script.js similarity index 100% rename from dashboard/script.js rename to q1-deploy-monitor/script.js diff --git a/dashboard/style.css b/q1-deploy-monitor/style.css similarity index 100% rename from dashboard/style.css rename to q1-deploy-monitor/style.css diff --git a/postmortem/postmortem.md b/q2-postmortem/postmortem-report.md similarity index 100% rename from postmortem/postmortem.md rename to q2-postmortem/postmortem-report.md diff --git a/.github/workflows/ci.yml b/q3-cicd/workflow.yml similarity index 100% rename from .github/workflows/ci.yml rename to q3-cicd/workflow.yml diff --git a/architecture/diagram.mmd b/q4-scalability/architecture.mmd similarity index 100% rename from architecture/diagram.mmd rename to q4-scalability/architecture.mmd diff --git a/q4-scalability/architecture.png b/q4-scalability/architecture.png new file mode 100644 index 0000000..e69de29 diff --git a/architecture/capacity.md b/q4-scalability/calculations.md similarity index 100% rename from architecture/capacity.md rename to q4-scalability/calculations.md diff --git a/mithal_monitor/dashboard/index.html b/q5-mithal-monitor/dashboard.html similarity index 100% rename from mithal_monitor/dashboard/index.html rename to q5-mithal-monitor/dashboard.html diff --git a/mithal_monitor/monitor.py b/q5-mithal-monitor/monitor.py similarity index 100% rename from mithal_monitor/monitor.py rename to q5-mithal-monitor/monitor.py diff --git a/scaling/auto_scaling_policy.md b/scaling/auto_scaling_policy.md deleted file mode 100644 index e5ae81a..0000000 --- a/scaling/auto_scaling_policy.md +++ /dev/null @@ -1,25 +0,0 @@ -# Auto-Scaling Policy for ghaymah.systems - -To prevent repeating the OOMKilled outage, the platform's auto-scaling group (ASG) must be configured to respond to memory pressure as well as CPU load. - -## 1. Scale-Out Policy (Adding Instances) -- **Metric:** Average Container Memory Utilization -- **Threshold:** > 70% -- **Evaluation Period:** 2 minutes (2 consecutive data points of 1 minute each) -- **Action:** Add 1 container instance (Step scaling) or scale by 20% of current capacity. -- **Cooldown Period:** 3 minutes (allows the new container to boot and start serving traffic before evaluating again). - -## 2. Scale-In Policy (Removing Instances) -- **Metric:** Average Container Memory Utilization -- **Threshold:** < 40% -- **Evaluation Period:** 5 minutes -- **Action:** Remove 1 container instance. -- **Cooldown Period:** 5 minutes (prevents aggressive scale-in which might cause immediate resource pressure). - -## 3. CPU Backup Policy -*Maintain existing CPU policies as a secondary trigger:* -- Scale out if Average CPU > 75% for 2 minutes. - -## 4. Minimum / Maximum Capacity -- **Min Containers:** 2 (for high availability across zones) -- **Max Containers:** 20 (to control billing, can be adjusted based on anticipated load) diff --git a/scaling/early_detection.md b/scaling/early_detection.md deleted file mode 100644 index 94ab6bd..0000000 --- a/scaling/early_detection.md +++ /dev/null @@ -1,25 +0,0 @@ -# Early Detection of Memory Issues - -Waiting for an application to crash (OOMKilled) is a reactive approach. To proactively detect memory issues, we must configure our monitoring tools (Prometheus, Datadog, or ghaymah metrics). - -## 1. High-Watermark Alerting -Configure alerts on the metric `container_memory_usage_bytes` (or equivalent). - -- **Warning Alert (Slack/Teams):** - - Trigger: Container Memory > 80% of limit - - Duration: Sustained for > 3 minutes. - - Action: Alerts the engineering team during business hours to investigate potential memory leaks. - -- **Critical Alert (PagerDuty/Phone Call):** - - Trigger: Container Memory > 90% of limit - - Duration: Sustained for > 2 minutes. - - Action: Wakes up the on-call engineer to apply mitigations (e.g., manual scaling, restarting pods) before the crash happens. - -## 2. Rate of Change Alerting (Anomaly Detection) -Sometimes memory doesn't hit a static threshold, but it grows unusually fast. -- Monitor the *derivative* (rate of change) of memory usage. -- If memory grows by more than 20% within 5 minutes (without a corresponding 20% spike in traffic), trigger an anomaly alert. - -## 3. APM Profiling -- Integrate APM (Application Performance Monitoring) to track Garbage Collection (GC) pauses in languages like Java/Node.js, or memory footprint per request in Python/Go. -- A sudden increase in GC time is often a precursor to an OOM event.