first commit

2026-07-27 23:27:12 +03:00
--- a/q2-postmortem/postmortem-report.md
+++ b/q2-postmortem/postmortem-report.md
@@ -0,0 +1,161 @@
+# Postmortem Report – Repeated OOMKilled Incident
+
+**Incident Date:** YYYY-MM-DD  
+**Duration:** 45 minutes  
+**Service:** Application hosted on Ghaymah Systems  
+**Severity:** High
+
+---
+
+# 1. Summary
+
+The application experienced repeated `OOMKilled` events, causing continuous container restarts and making the service unavailable for approximately 45 minutes.
+
+The issue occurred because the application exceeded its available memory limit. Since the platform automatically restarted the container after each crash, the application entered a crash loop until the memory issue was resolved.
+
+---
+
+# 2. Timeline
+
+| Time | Event |
+|------|-------|
+| 10:00 | New application version deployed |
+| 10:05 | Memory usage started increasing rapidly |
+| 10:10 | First `OOMKilled` event occurred |
+| 10:11 | Platform restarted the container |
+| 10:14 | Container exceeded memory limit again |
+| 10:15 | Second `OOMKilled` event |
+| 10:20 | Multiple restart attempts continued |
+| 10:35 | Engineering team identified abnormal memory consumption |
+| 10:45 | Memory issue resolved and application recovered |
+
+---
+
+# 3. Root Cause Analysis
+
+## Immediate Cause
+
+The application consumed more memory than the container's configured memory limit, causing the Linux Out-Of-Memory (OOM) Killer to terminate the process.
+
+## Root Cause
+
+Possible contributing factors include:
+
+- Memory leak in the application
+- Large objects remaining in memory
+- Insufficient memory limits for production workload
+- Lack of early monitoring and alerting
+
+---
+
+# 4. Impact
+
+- Service unavailable for 45 minutes
+- Users could not access the application
+- Multiple container restarts
+- Increased error rate and failed requests
+
+---
+
+# 5. Recommendations
+
+## Short-Term
+
+- Increase container memory limit.
+- Restart affected containers.
+- Verify application memory usage after deployment.
+- Roll back if abnormal memory growth is detected.
+
+## Long-Term
+
+- Fix memory leaks.
+- Perform load testing before production deployments.
+- Configure monitoring and alerting for memory usage.
+- Monitor container restart count.
+- Enable automatic scaling.
+- Establish deployment health checks.
+
+---
+
+# Auto-Scaling Policy
+
+## Objective
+
+Prevent service outages caused by high resource utilization.
+
+### Scale-Out Rules
+
+- Add one new instance when:
+  - CPU usage > 70% for 5 minutes.
+  - Memory usage > 80% for 5 minutes.
+  - Average response time > 500 ms.
+
+### Scale-In Rules
+
+- Remove one instance when:
+  - CPU usage < 30% for 10 minutes.
+  - Memory usage < 40% for 10 minutes.
+
+### Minimum Instances
+
+- 2 running instances
+
+### Maximum Instances
+
+- 10 running instances
+
+### Health Checks
+
+- Check `/health` every 30 seconds.
+- Replace unhealthy containers automatically.
+
+---
+
+# Early Detection Using Monitoring
+
+To detect similar issues before they cause downtime, monitor the following metrics:
+
+## Infrastructure Metrics
+
+- Memory Usage
+- Memory Limit
+- CPU Usage
+- Container Restarts
+- OOMKilled Events
+- Disk Usage
+
+## Application Metrics
+
+- HTTP Response Time
+- Request Rate
+- Error Rate (4xx / 5xx)
+- Active Connections
+
+## Alerts
+
+Create alerts when:
+
+- Memory usage exceeds 80%
+- Container restart count increases
+- OOMKilled event detected
+- Response time exceeds 500 ms
+- Error rate exceeds 5%
+
+## Monitoring Stack
+
+Example monitoring solution:
+
+- Prometheus (metrics collection)
+- Grafana (dashboards)
+- Alertmanager (notifications)
+- Email / Slack / Microsoft Teams notifications
+
+---
+
+# Lessons Learned
+
+- Resource limits should be validated before deployment.
+- Memory consumption should be continuously monitored.
+- Health checks and alerts must be configured for production services.
+- Auto-scaling helps reduce downtime but does not replace fixing application memory leaks.
+- Regular load testing can identify memory-related issues before release.
--- a/q2-postmortem/timeline.svg
+++ b/q2-postmortem/timeline.svg
@@ -0,0 +1,70 @@
+<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 900 380" font-family="Segoe UI, Arial, sans-serif">
+  <defs>
+    <linearGradient id="pbg" x1="0%" y1="0%" x2="0%" y2="100%">
+      <stop offset="0%" style="stop-color:#0f172a"/>
+      <stop offset="100%" style="stop-color:#1e293b"/>
+    </linearGradient>
+    <filter id="psh" x="-10%" y="-10%" width="120%" height="130%">
+      <feDropShadow dx="0" dy="3" stdDeviation="5" flood-opacity="0.3"/>
+    </filter>
+  </defs>
+
+  <rect width="900" height="380" fill="url(#pbg)" rx="12"/>
+
+  <!-- Title -->
+  <text x="450" y="38" text-anchor="middle" fill="#e2e8f0" font-size="18" font-weight="700">Q2: OOMKilled Incident — Postmortem Timeline</text>
+  <text x="450" y="58" text-anchor="middle" fill="#94a3b8" font-size="11">45-minute outage caused by repeated memory limit breaches</text>
+
+  <!-- Timeline line -->
+  <line x1="60" y1="100" x2="60" y2="340" stroke="#334155" stroke-width="3"/>
+
+  <!-- Timeline nodes -->
+  <!-- 10:00 - Deploy -->
+  <circle cx="60" cy="110" r="8" fill="#3b82f6"/>
+  <text x="80" y="108" fill="#3b82f6" font-size="11" font-weight="700">10:00</text>
+  <text x="80" y="124" fill="#e2e8f0" font-size="10">New version deployed</text>
+  <text x="80" y="138" fill="#94a3b8" font-size="9">Deploy triggered normally</text>
+
+  <!-- 10:05 - Memory spike -->
+  <circle cx="60" cy="165" r="8" fill="#eab308"/>
+  <text x="80" y="163" fill="#eab308" font-size="11" font-weight="700">10:05</text>
+  <text x="80" y="179" fill="#e2e8f0" font-size="10">Memory usage spikes</text>
+  <text x="80" y="193" fill="#94a3b8" font-size="9">Rapid memory consumption detected</text>
+
+  <!-- 10:10 - First OOM -->
+  <circle cx="60" cy="225" r="8" fill="#ef4444"/>
+  <text x="80" y="223" fill="#ef4444" font-size="11" font-weight="700">10:10</text>
+  <text x="80" y="239" fill="#e2e8f0" font-size="10">First OOMKilled event</text>
+  <text x="80" y="253" fill="#94a3b8" font-size="9">Container killed by Linux OOM Killer</text>
+
+  <!-- 10:11 - Restart loop -->
+  <circle cx="60" cy="280" r="8" fill="#ef4444"/>
+  <text x="80" y="278" fill="#ef4444" font-size="11" font-weight="700">10:11 - 10:35</text>
+  <text x="80" y="294" fill="#e2e8f0" font-size="10">Crash loop continues</text>
+  <text x="80" y="308" fill="#94a3b8" font-size="9">Multiple restart attempts, each hitting memory limit</text>
+
+  <!-- 10:35 - Team investigates -->
+  <circle cx="60" cy="335" r="8" fill="#eab308"/>
+  <text x="80" y="333" fill="#eab308" font-size="11" font-weight="700">10:35</text>
+  <text x="80" y="349" fill="#e2e8f0" font-size="10">Team identifies root cause</text>
+  <text x="80" y="363" fill="#94a3b8" font-size="9">Memory leak found, fix deployed</text>
+
+  <!-- Right side: Root Cause Box -->
+  <rect x="500" y="85" width="380" height="130" rx="10" fill="#1e293b" stroke="#ef4444" stroke-width="1.5" filter="url(#psh)"/>
+  <text x="520" y="112" fill="#ef4444" font-size="12" font-weight="700">Root Cause</text>
+  <line x1="520" y1="122" x2="860" y2="122" stroke="#334155" stroke-width="0.5"/>
+  <text x="520" y="142" fill="#94a3b8" font-size="10">• Application exceeded memory limit</text>
+  <text x="520" y="160" fill="#94a3b8" font-size="10">• Memory leak in new version</text>
+  <text x="520" y="178" fill="#94a3b8" font-size="10">• No memory monitoring/alerting</text>
+  <text x="520" y="196" fill="#94a3b8" font-size="10">• Auto-restart created crash loop</text>
+
+  <!-- Right side: Impact Box -->
+  <rect x="500" y="230" width="380" height="80" rx="10" fill="#1e293b" stroke="#f97316" stroke-width="1.5" filter="url(#psh)"/>
+  <text x="520" y="257" fill="#f97316" font-size="12" font-weight="700">Impact</text>
+  <line x1="520" y1="267" x2="860" y2="267" stroke="#334155" stroke-width="0.5"/>
+  <text x="520" y="287" fill="#94a3b8" font-size="10">45 min downtime • 100% request failure • 12+ restarts</text>
+
+  <!-- Right side: Fix Box -->
+  <rect x="500" y="325" width="380" height="40" rx="10" fill="#1e293b" stroke="#22c55e" stroke-width="1.5" filter="url(#psh)"/>
+  <text x="690" y="350" text-anchor="middle" fill="#22c55e" font-size="11" font-weight="600">Fixed: Memory leak patched + limits increased</text>
+</svg>