68 أسطر
2.2 KiB
Markdown
68 أسطر
2.2 KiB
Markdown
# Ghaymah Auto-Scaling Policy
|
|
|
|
## Goal
|
|
|
|
Design an auto-scaling policy for the Ghaymah platform that reduces the impact of memory-related incidents (like the OOMKilled outage in the postmortem), while making clear that auto-scaling alone does not fix a memory leak — it only limits the damage.
|
|
|
|
---
|
|
|
|
## Important Note
|
|
|
|
The original incident was caused by a **memory leak**, not by a real traffic increase. Auto-scaling can't fix a leak — a new pod created by auto-scaling will eventually leak too. So this policy is a **safety net**, not a replacement for fixing the leak itself.
|
|
|
|
---
|
|
|
|
## 1. Right-Size Requests and Limits First
|
|
|
|
Before any auto-scaling works well, the memory `requests` and `limits` need to be based on real usage instead of a guess. Otherwise the autoscaler is scaling based on wrong numbers.
|
|
|
|
---
|
|
|
|
## 2. Horizontal Pod Autoscaler (HPA) — based on memory
|
|
|
|
Scale the number of pods when average memory usage crosses a threshold, so no single pod gets close to its limit as fast.
|
|
|
|
```yaml
|
|
apiVersion: autoscaling/v2
|
|
kind: HorizontalPodAutoscaler
|
|
metadata:
|
|
name: ghaymah-app-hpa
|
|
spec:
|
|
scaleTargetRef:
|
|
apiVersion: apps/v1
|
|
kind: Deployment
|
|
name: ghaymah-app
|
|
minReplicas: 2
|
|
maxReplicas: 6
|
|
metrics:
|
|
- type: Resource
|
|
resource:
|
|
name: memory
|
|
target:
|
|
type: Utilization
|
|
averageUtilization: 70
|
|
```
|
|
|
|
---
|
|
|
|
## 3. Cluster Autoscaler
|
|
|
|
If HPA creates more pods than the current nodes can fit, the Cluster Autoscaler adds more nodes automatically so pods aren't stuck pending.
|
|
|
|
---
|
|
|
|
## 4. Alerting Before Scaling Limits Are Hit
|
|
|
|
Add an alert when memory usage is trending up (not just when it hits the limit), so the team can catch a leak early instead of waiting for auto-scaling or a crash.
|
|
|
|
---
|
|
|
|
## 5. Restart / Readiness Behavior
|
|
|
|
Make sure the app has proper readiness probes, so traffic isn't sent to a pod that just restarted and isn't ready yet — this reduces user-facing errors during restart loops.
|
|
|
|
---
|
|
|
|
## Summary
|
|
|
|
This policy helps absorb real load spikes and buys extra time before hitting memory limits, which reduces how often OOMKilled events happen. But it does **not** replace fixing the actual memory leak — that's still the real fix. |