# v2 git bundle 5419a512653611fdce84290c40b13b48b3687ee5 refs/heads/bundle 5419a512653611fdce84290c40b13b48b3687ee5 HEAD PACK% x,tree 058b78371a352c276d49dc0c309fb555004c20a5 parent 4fd4bb72e10d210a5921ea8099bb1d35dab2d878 author mac 1785149530 +0300 committer mac 1785149530 +0300 first commit P=Wxtree 180c983537969993b83c73873b2dab2943bc52dc author mac 1785149084 +0300 committer mac 1785149084 +0300 feat: complete initial project structure for API and Monitoring Dashboards Fxz40000 .githubN 4X8zA K? 100644 .gitignore7OA~:z+wK100644 README.mdm,3uHRS@Ր40000 api:䞅6}+tTx340000 architecture2I;ϬP40000 civ-hI.  #H40000 dashboardb,019UDia40000 mithal_monitor`N(wg`ÍqI40000 monitoreBqevqH&lǾ40000 postmortemw._3HA3pp40000 scalingH0\rH1*x !bx$40000 workflowsD;R]_` GfKx"100644 ci.yml tPpsI?̳f xxTMo@+FiTS\V㴲CNv~8 +iJ X7oVCby)\`U5 Y%KyR@eub"Q_> CrGh<],!(̸6jΆj8Bx7HAfU峊{OoOBll~ jEN!=1@׭xK B&|+r)3ࢧ0*72C5(dƅ_8_}lIVZ 7*K⨸18Bdx=Nn~,y7bZKY~]=Ν[pd»a+%'^Υ?d`B=n`!Gul:ik~z:g t ~S)uصk\IQ MӶ6p mZG\Y4?If 8z+at9%˒؅mR~=忠0բVNJn|#4Hf-oW21aYt(nyf aunBc !Α7r+!wS:eCE^uK_DU{7gW4xKvenv/ __pycache__/ *.pyc .env monitor.log api.log monitor_sh.log .DS_Store '>x# SRE-project B\xt100644 Dockerfile5eIcΨxT)<100644 app.py}HHBEslw?ptB100644 requirements.txtfYlK"k0 xs U(,ϳ234- v RO,(rT(J-,,JM+)+( S(,P+.IQMNLHM,R-6h^A%rvU00ruQV8NIGA D) ̈< x5 Jc=5Jl]ݛ2s9|3~u\{+UDFG[ߕ)y]x^7ǭ Vň"ⷑNjƠ$6$"yԔ+w f^==x Flask==3.0.3 |h x\100644 block_storage.mdի1D~jܦ 100644 capacity.mdnL&Iڐ[9<'4q100644 cold_start.md5|bǘ^100644 diagram.mmd7Etd(_kt#7@жXxy# Using ghaymah Block Storage for Stateful Workloads Containers are inherently stateless; they lose their local filesystem data when they are destroyed or rescheduled. For applications that require persistent data (Stateful Workloads), we utilize **ghaymah Block Storage**. ## 1. What is Block Storage? Block Storage provides persistent, highly available disk volumes that can be attached to containers. Unlike object storage (S3), it behaves like a physical hard drive mounted to the OS. ## 2. Use Cases in our Architecture While our API application is mostly stateless, certain workloads require persistence: - **Local Caching:** If a container downloads large datasets or machine learning models upon startup, these can be stored on Block Storage so subsequent container restarts are faster. - **Session Data / Logs:** If we are writing complex audit logs that haven't yet been shipped to a centralized logging service. - **Databases:** If running a self-managed database (e.g., PostgreSQL or Redis) within a container, Block Storage is mandatory to prevent data loss. ## 3. Configuration & Mounting When deploying via the `ghaymah deploy` CLI or dashboard, we specify a volume mount: ```yaml volumes: - name: my-persistent-data size: 50GB mountPath: /mnt/data ``` Inside the container, the application can simply read/write files to `/mnt/data/` knowing the data will survive container restarts. IFx}S͎0)FZ!unZ$";I&2v^'an=biZtθ#|FWY|8LweK8vY54 c1nc!G]E X ,VvJ!l ֑hh1ҨׅL 4N~〮$g |@&xm0"A/[Q/1rt wgqyo$SOin&o͋#g5eS]SĆ`g ~eA&_Ov3dy$,+;Vht^rRMD`AL} c,rLalr `6?ndٽ ZŸ(Qf#W$!Ү\ʾKrp`u]g_p?~;3o{{dC/r1E.E;,5%^^B=C@[h;6^\=$cZ0mZ JY[!AODn#K&CPC'` cxCe}WǪ"ú6%pk^ww-Sj[0M` ~tJt7YE}}BHy.bh-k 7s Yxn# Cold Start Strategy When auto-scaling responds to a traffic spike, new containers must be initialized. The time it takes from the scaling decision to the container actually serving requests is the "cold start" latency. To minimize this delay and prevent dropped requests, we implement the following strategy: ## 1. Lightweight Base Images - Use Alpine or distroless base images (e.g., `python:3.11-alpine`). - Smaller images pull faster from the Container Registry over the network. ## 2. Pre-warming (Buffer Pool) - Configure the Auto-Scaling Group to always maintain a "buffer" of idle containers (e.g., 10% of the current required capacity). - If we need 43 containers for active load, we run ~47 containers. When traffic spikes, these 4 idle containers can serve requests instantly while the ASG provisions new ones. ## 3. Lazy Loading & Readiness Probes - Defer non-critical initialization (like building large in-memory caches) until *after* the container has started accepting requests. - Configure Kubernetes/ghaymah readiness probes to accurately reflect when the app is ready to serve traffic, ensuring the load balancer doesn't route traffic to a container that is still booting. ## 4. Keep-Alive & Connection Pooling - Ensure idle containers aren't prematurely terminated. Keep database connections alive in a connection pool to avoid the latency of establishing new TCP handshakes during a sudden burst. =x*```mermaid graph TD Client((Client Requests
15,000 req/s)) --> WAF[Web Application Firewall] WAF --> LB[ghaymah Load Balancer] subgraph Auto-Scaling Group direction LR LB -->|Traffic Distribution| API1[myapp-api Container 1
~350 req/s] LB --> API2[myapp-api Container 2
~350 req/s] LB --> API3[myapp-api Container 3] LB -.-> APIN[myapp-api Container N
Total: 43 Containers] end API1 --> BS1[(ghaymah Block Storage
/mnt/data)] API2 --> BS2[(ghaymah Block Storage
/mnt/data)] API3 --> BS3[(ghaymah Block Storage
/mnt/data)] APIN --> BSN[(ghaymah Block Storage
/mnt/data)] classDef container fill:#e3f2fd,stroke:#1565c0,stroke-width:2px; classDef lb fill:#fff3e0,stroke:#e65100,stroke-width:2px; classDef storage fill:#e8f5e9,stroke:#2e7d32,stroke-width:2px; class API1,API2,API3,APIN container; class LB lb; class BS1,BS2,BS3,BSN storage; ``` 3(MxO100644 README.mdڷGvwdky100644 ghaymah_cli.mdQ-Xxj+ofe` TU_ ]x## Environments: Staging vs Production In our deployment pipeline, we utilize two main environments: **Staging** and **Production**. Understanding the differences between them is crucial for safe software delivery. ## 1. Staging Environment (`staging`) - **Purpose:** A pre-production area for QA testing, integration testing, and final review by stakeholders before a release goes live. - **Data:** Uses dummy, sanitized, or replicated data. NEVER connects to the live production database. - **Access:** Restricted to internal team members, developers, and QA testers. Often protected by VPN, IP whitelisting, or Basic Auth. - **Scale:** Typically scaled down (fewer containers, smaller database instances) to save costs, as it doesn't need to handle user traffic. - **Deployment:** Automatic upon merging code to the `main` branch. ## 2. Production Environment (`production`) - **Purpose:** The live environment that real users interact with. - **Data:** Contains live, sensitive, real user data. Strict access controls and backups are enforced. - **Access:** Publicly accessible (for web apps/APIs). Infrastructure access is strictly limited to authorized SREs/DevOps personnel. - **Scale:** Scaled up to handle expected user load, with Auto-Scaling policies enabled to handle traffic spikes. - **Deployment:** Requires a **Manual Approval** step in the CI/CD pipeline (e.g., in GitHub Actions) to ensure that the code deployed to staging has been properly vetted and approved for live release.  ͶLx9# Integrating with the ghaymah CLI To manage and deploy applications to ghaymah.systems from your local machine or CI/CD pipeline, you need to use the `ghaymah` CLI. ## 1. Installation Depending on your OS, install the CLI (example for macOS/Linux): ```bash curl -sL https://cli.ghaymah.systems/install.sh | bash ``` ## 2. Authentication Log in to your ghaymah account: ```bash ghaymah login ``` This will open a browser window to authenticate. If you are in a CI/CD environment (headless), use a token: ```bash ghaymah login --token $GHAYMAH_TOKEN ``` ## 3. Pushing Images to ghaymah Container Registry Authenticate Docker with the ghaymah registry: ```bash docker login registry.ghaymah.systems -u $GHAYMAH_USERNAME -p $GHAYMAH_TOKEN ``` Build and push your image: ```bash docker build -t registry.ghaymah.systems/my-org/myapp-api:v1 . docker push registry.ghaymah.systems/my-org/myapp-api:v1 ``` ## 4. Deploying the Application Once the image is in the registry, deploy it using the CLI: ```bash ghaymah deploy \ --name myapp-api \ --image registry.ghaymah.systems/my-org/myapp-api:v1 \ --port 8080 \ --env production ``` You can also monitor logs in real-time: ```bash ghaymah logs myapp-api --follow ``` Txp100644 index.html+V"gޒ'qu100644 script.js;p9\6Kk100644 style.cssSɿK-x"k6xc/n@x Ghaymah API Monitoring

نظام المراقبة السحابي (API)

حالة الخدمة
جاري الفحص...
زمن الاستجابة
- ms
إجمالي الطلبات
0
pjoMx!let requestCount = 0; async function checkHealth() { const startTime = Date.now(); try { const response = await fetch('http://localhost:8080/health'); const latency = Date.now() - startTime; requestCount++; if (response.ok) { document.getElementById('status').innerHTML = ` متصل ويعمل `; } else { document.getElementById('status').innerHTML = ` خطأ بالخادم `; } document.getElementById('latency').innerHTML = `${latency} ms`; document.getElementById('requests').innerHTML = requestCount.toLocaleString(); } catch (error) { document.getElementById('status').innerHTML = ` غير متصل `; document.getElementById('latency').innerHTML = `- ms`; } } setInterval(checkHealth, 5000); checkHealth(); Uƶxv @import url('https://fonts.googleapis.com/css2?family=Outfit:wght@300;400;600;700&display=swap'); :root { --bg-color: #0f172a; --card-bg: rgba(30, 41, 59, 0.7); --text-main: #f8fafc; --text-muted: #94a3b8; --accent: #3b82f6; --success: #10b981; --danger: #ef4444; --glow-success: rgba(16, 185, 129, 0.4); --glow-danger: rgba(239, 68, 68, 0.4); } body { font-family: 'Outfit', sans-serif; background-color: var(--bg-color); background-image: radial-gradient(at 0% 0%, rgba(59, 130, 246, 0.15) 0px, transparent 50%), radial-gradient(at 100% 100%, rgba(139, 92, 246, 0.15) 0px, transparent 50%); color: var(--text-main); text-align: center; min-height: 100vh; margin: 0; display: flex; flex-direction: column; align-items: center; justify-content: center; padding: 20px; } h1 { font-size: 3rem; font-weight: 700; margin-bottom: 40px; background: linear-gradient(to right, #60a5fa, #c084fc); -webkit-background-clip: text; -webkit-text-fill-color: transparent; letter-spacing: 2px; } .dashboard-container { display: flex; gap: 20px; flex-wrap: wrap; justify-content: center; max-width: 900px; } .card { background: var(--card-bg); backdrop-filter: blur(12px); -webkit-backdrop-filter: blur(12px); border: 1px solid rgba(255, 255, 255, 0.1); padding: 30px; border-radius: 20px; min-width: 250px; box-shadow: 0 10px 30px -10px rgba(0, 0, 0, 0.5); transition: transform 0.3s ease, box-shadow 0.3s ease; position: relative; overflow: hidden; } .card:hover { transform: translateY(-5px); box-shadow: 0 20px 40px -10px rgba(0, 0, 0, 0.6); border: 1px solid rgba(255, 255, 255, 0.2); } .card::before { content: ''; position: absolute; top: 0; left: 0; right: 0; height: 3px; background: linear-gradient(to right, transparent, var(--accent), transparent); opacity: 0.5; } .card-title { font-size: 1.1rem; color: var(--text-muted); margin-bottom: 15px; text-transform: uppercase; letter-spacing: 1px; } .card-value { font-size: 2.5rem; font-weight: 600; display: flex; align-items: center; justify-content: center; gap: 10px; } .good { color: var(--success); text-shadow: 0 0 15px var(--glow-success); } .bad { color: var(--danger); text-shadow: 0 0 15px var(--glow-danger); } .unknown { color: var(--text-muted); } /* Ping Animation for Status */ .status-indicator { display: inline-block; width: 15px; height: 15px; border-radius: 50%; position: relative; } .status-indicator.good { background-color: var(--success); } .status-indicator.bad { background-color: var(--danger); } .status-indicator.good::after { content: ''; position: absolute; width: 100%; height: 100%; top: 0; left: 0; border-radius: 50%; background-color: var(--success); animation: ping 2s cubic-bezier(0, 0, 0.2, 1) infinite; } @keyframes ping { 75%, 100% { transform: scale(2.5); opacity: 0; } } ,ˑsx|40000 dashboard-ZbzdD0-n100644 deploy_instructions.mddvUBrDIve%)r100644 monitor.py5=aUqA1Tx&100644 index.htmla4B k˵mcYcxێ=_1aREJkn-j'Qot(@FH7 W5h;5 })_7pݠ"E? )]h^hos=NS2Sa0`H@Р >CC#F>)Jq3kG 9#b頙gZyA3yOF>#% |r +I,+x R5&\^6=)[АS Ӡ7/߆ߛs$C9A T3J ("aH峀 ;b%2 Ҋ8V;~,k<푏0fzT="cZsnqn_T\.kA0U ?:llwepy,B؟tA'%#L֤{ӽ閩c㱖逈c;8/JO=b2DHZ ><ޣ<XAJV G> }i;9ict@GsXS y\xT'DpFfh J*  ,?4% mрOANt%2ρ60S_^S<',=,`` FS[< y '9'^&̚ ޹5羚rI4=ac엤u;)VqAnz$ciSP=DB|n9J%s?AH8 %X` "89{ ћgL`xsTtW5@5å2A2mKbPC+XX<]QvEesiƁn =iShN[P 蜰8D)8mL%݈n)\2e}Zbmi'BrH^5Ѭӝ+ *:FoE*eCHcZ;jO-**KSUd%xn ;gz=E.xDZw :IgVpɴmkL) "Jf:VZL޿:ej@ߐ fg ̻;Z>Y>[Y"˧o/VfHψP){cӊPQqV/;dxx BXq'>hb)T+[G4%4ZaV<9*Q/TZXY>G|''FT^aJSO _^{ ~JCz6𪉼JdQzVr'RKI|EЮ]w⯋C|:dZ2U9JΨJ 3]F>R:͈NfV^cӿ*WH/1(I,nd`k+Ne+i{/ߌ|g Gi_7>_'tˡ_x7' D,v 0T+6Pzq n7.jLYHឯv`P,^f mh"++)]7p&9GħD|ƑnV1!3$ &yڜ8SF\,I{4.&ia{M E~]^jH@,dH<j!%DlN4kŇU`ͧH (SN-?9tSQvYG;q*$ fτ<l*>QФ(lVYgٓ F?'v7eI/ RAg l|YlK#. onpJ:?mip~,VvǷ\-߬" Ay>k/>E^ _n{Fgvn-*V0._{˛+T['pM Y9u>dG#1j t"All?BY=f +=إenܥDe5Ph Cp+^Ew$ѪIbsyVj SV}͂cw.n68)*L} ! S}\mFxH.*` fIԉ#Ysdk=𲘅P]9/0euúTݲU*3[jƍ]o ץc^g Uc񇗺ɨ^zT|+hZmu|iyWѝ{`H4K5Π5L0L\4cQx# Deployment Instructions: Mithal Monitoring Dashboard To serve the HTML dashboard and the collected `metrics.json` data, we will deploy a simple Nginx container to ghaymah.systems. ## 1. Directory Structure Setup Move the dashboard HTML and the `metrics.json` file into a single directory to be served: ```bash mkdir -p /Users/mac/a1/mithal_monitor/deploy/public cp /Users/mac/a1/mithal_monitor/dashboard/index.html /Users/mac/a1/mithal_monitor/deploy/public/ # Note: The monitor.py script should be configured to write metrics.json to this 'public' folder. ``` ## 2. Nginx Dockerfile Create a `Dockerfile` in `/Users/mac/a1/mithal_monitor/deploy/`: ```dockerfile FROM nginx:alpine COPY public/ /usr/share/nginx/html/ EXPOSE 80 ``` ## 3. Deployment Steps Using the `ghaymah` CLI: ```bash cd /Users/mac/a1/mithal_monitor/deploy # Build and Push docker build -t registry.ghaymah.systems/my-org/mithal-dashboard:latest . docker push registry.ghaymah.systems/my-org/mithal-dashboard:latest # Deploy ghaymah deploy \ --name mithal-dashboard \ --image registry.ghaymah.systems/my-org/mithal-dashboard:latest \ --port 80 \ --env production ``` The dashboard will now be accessible via the URL provided by ghaymah.systems, and it will serve `index.html` as well as `metrics.json` over HTTP(S). ʹxVmo6_q`D}0f ƒtktHJRu }Gu>ؔJ* ?֨%k-4;]˿Z`d 7h?o[LD'TAZrn]1Fen6HtS [׋7m|X-W4ըO<+ȍT#*J4*Oubk 5Le么gЕJa!X2%Bꄑ? ü1QMsFRD>YfC \h%[+ Go-6RK1 UYh*DeN~rRD6Ȇݝ ik~/d0x5\K 1r5Shkλ\IFqؐ#V?xg7hH2ph,5RR0~&q_$}uhLG )pG^Ob'Y|XȢO1| x<;4u2Hʶ]DhsQ gېZ*9}8" #+5Oz3 C2{)~.XFV$dlfЎo?aNJYtD늛MB7v6/&+UcSdžHHݻZBWrPt4pՙ;#c~!$YX8ۛۛ zj=, K7 Lq34D/5 %#is6rx{Ա!*PCX! 3'58cv9}\1f+c cmj4(q%$Ԧx&100755 monitor.sh^ڙM6:I كYexUON0 +LHLdTI LZD.UR&UnHP'g???O.x79^xEfX=7ْWV\֚&X7km?)!ȌBhn|:Ȉ#錒V;հ\w -(yVn=D@Hz-amdɞRx(~ɊȢCbKb 1o&)5bs"Akx)100644 postmortem.mdnJRPKV?֖{<+xmU]O#G|h,llD@],Cn;f>9>ճ\#.Fzp!Nj#WtpG_utSTh>TGV;z5_ǝ #WQ;+o/Tk"nTT ]՝ T1)YQ+ôv΅SnH[R4>{^/ftJvjOg] ʞ@OL^pb|8ac cg@z]l%a* 3W,\xӕNU*!f=*E7 Xơ~ǕVls}x>3` \(b ^F*u۰w:/~b"˾l *;<,vRѐ LƸyƻTcq";eMZ 8'qWbK?ҧǓ{ 9(V'%Z.l6XZE%5 Wwʷ=ib|c _E+]󛳛Cvл\- _9Ϡx`100644 auto_scaling_policy.md宁.!^87P_ٰ100644 early_detection.mdkӦZ&qN^u^,&7Lx7# Auto-Scaling Policy for ghaymah.systems To prevent repeating the OOMKilled outage, the platform's auto-scaling group (ASG) must be configured to respond to memory pressure as well as CPU load. ## 1. Scale-Out Policy (Adding Instances) - **Metric:** Average Container Memory Utilization - **Threshold:** > 70% - **Evaluation Period:** 2 minutes (2 consecutive data points of 1 minute each) - **Action:** Add 1 container instance (Step scaling) or scale by 20% of current capacity. - **Cooldown Period:** 3 minutes (allows the new container to boot and start serving traffic before evaluating again). ## 2. Scale-In Policy (Removing Instances) - **Metric:** Average Container Memory Utilization - **Threshold:** < 40% - **Evaluation Period:** 5 minutes - **Action:** Remove 1 container instance. - **Cooldown Period:** 5 minutes (prevents aggressive scale-in which might cause immediate resource pressure). ## 3. CPU Backup Policy *Maintain existing CPU policies as a secondary trigger:* - Scale out if Average CPU > 75% for 2 minutes. ## 4. Minimum / Maximum Capacity - **Min Containers:** 2 (for high availability across zones) - **Max Containers:** 20 (to control billing, can be adjusted based on anticipated load) -Wxv# Early Detection of Memory Issues Waiting for an application to crash (OOMKilled) is a reactive approach. To proactively detect memory issues, we must configure our monitoring tools (Prometheus, Datadog, or ghaymah metrics). ## 1. High-Watermark Alerting Configure alerts on the metric `container_memory_usage_bytes` (or equivalent). - **Warning Alert (Slack/Teams):** - Trigger: Container Memory > 80% of limit - Duration: Sustained for > 3 minutes. - Action: Alerts the engineering team during business hours to investigate potential memory leaks. - **Critical Alert (PagerDuty/Phone Call):** - Trigger: Container Memory > 90% of limit - Duration: Sustained for > 2 minutes. - Action: Wakes up the on-call engineer to apply mitigations (e.g., manual scaling, restarting pods) before the crash happens. ## 2. Rate of Change Alerting (Anomaly Detection) Sometimes memory doesn't hit a static threshold, but it grows unusually fast. - Monitor the *derivative* (rate of change) of memory usage. - If memory grows by more than 20% within 5 minutes (without a corresponding 20% spike in traffic), trigger an anomaly alert. ## 3. APM Profiling - Integrate APM (Application Performance Monitoring) to track Garbage Collection (GC) pauses in languages like Java/Node.js, or memory footprint per request in Python/Go. - A sudden increase in GC time is often a precursor to an OOM event. Jj_x Hmizb{{hroǿv5v