26 أسطر
1.4 KiB
Markdown
26 أسطر
1.4 KiB
Markdown
# Early Detection of Memory Issues
|
|
|
|
Waiting for an application to crash (OOMKilled) is a reactive approach. To proactively detect memory issues, we must configure our monitoring tools (Prometheus, Datadog, or ghaymah metrics).
|
|
|
|
## 1. High-Watermark Alerting
|
|
Configure alerts on the metric `container_memory_usage_bytes` (or equivalent).
|
|
|
|
- **Warning Alert (Slack/Teams):**
|
|
- Trigger: Container Memory > 80% of limit
|
|
- Duration: Sustained for > 3 minutes.
|
|
- Action: Alerts the engineering team during business hours to investigate potential memory leaks.
|
|
|
|
- **Critical Alert (PagerDuty/Phone Call):**
|
|
- Trigger: Container Memory > 90% of limit
|
|
- Duration: Sustained for > 2 minutes.
|
|
- Action: Wakes up the on-call engineer to apply mitigations (e.g., manual scaling, restarting pods) before the crash happens.
|
|
|
|
## 2. Rate of Change Alerting (Anomaly Detection)
|
|
Sometimes memory doesn't hit a static threshold, but it grows unusually fast.
|
|
- Monitor the *derivative* (rate of change) of memory usage.
|
|
- If memory grows by more than 20% within 5 minutes (without a corresponding 20% spike in traffic), trigger an anomaly alert.
|
|
|
|
## 3. APM Profiling
|
|
- Integrate APM (Application Performance Monitoring) to track Garbage Collection (GC) pauses in languages like Java/Node.js, or memory footprint per request in Python/Go.
|
|
- A sudden increase in GC time is often a precursor to an OOM event.
|