1.5 KiB
Early Detection of the Memory Leak Using Ghaymah's Monitoring Tools
Idea
The problem should ideally be caught before memory hits the limit and the pod crashes, not after. This means watching how memory usage changes over time, not just checking if it already reached a critical point.
1. Watch Memory Usage Over Time (Prometheus)
Instead of only alerting when memory is almost at the limit, we can track memory usage per pod using Prometheus and set an alert for when it keeps increasing steadily over several minutes without going back down. That's usually a sign of a leak, and it can be noticed while memory is still at a safe level.
2. Use Grafana to Visualize It
A simple Grafana graph of memory usage per pod makes a leak easy to notice — instead of the usage going up and down normally, you'd see a line that keeps climbing until it crashes and restarts, then climbs again. That pattern is a clear sign something's wrong.
3. Track Restart Count
Kubernetes also exposes how many times a pod restarted. If a pod is restarting a lot in a short time, that alone is a signal to check what's going on, even before looking at memory numbers specifically.
Summary
Instead of just alerting "memory is almost full" (which is too late), the goal is to alert on the memory trend going up steadily and on repeated restarts. Using Prometheus + Grafana for this would have shown the problem building up before the first crash actually happened.