SRE-final-project/scaling/early_detection.md

Early Detection of Memory Issues

Waiting for an application to crash (OOMKilled) is a reactive approach. To proactively detect memory issues, we must configure our monitoring tools (Prometheus, Datadog, or ghaymah metrics).

1. High-Watermark Alerting

Configure alerts on the metric container_memory_usage_bytes (or equivalent).

• Warning Alert (Slack/Teams):

  • Trigger: Container Memory > 80% of limit
  • Duration: Sustained for > 3 minutes.
  • Action: Alerts the engineering team during business hours to investigate potential memory leaks.

• Critical Alert (PagerDuty/Phone Call):

  • Trigger: Container Memory > 90% of limit
  • Duration: Sustained for > 2 minutes.
  • Action: Wakes up the on-call engineer to apply mitigations (e.g., manual scaling, restarting pods) before the crash happens.

2. Rate of Change Alerting (Anomaly Detection)

Sometimes memory doesn't hit a static threshold, but it grows unusually fast.

• Monitor the derivative (rate of change) of memory usage.
• If memory grows by more than 20% within 5 minutes (without a corresponding 20% spike in traffic), trigger an anomaly alert.

3. APM Profiling

• Integrate APM (Application Performance Monitoring) to track Garbage Collection (GC) pauses in languages like Java/Node.js, or memory footprint per request in Python/Go.
• A sudden increase in GC time is often a precursor to an OOM event.