edited some files
هذا الالتزام موجود في:
@@ -70,4 +70,16 @@ The incident could have been caught well before the OOMKilled events with the fo
|
||||
- **Trend-based, not just threshold-based, alerts.** A memory-usage graph that climbs steadily after each deploy is a leading indicator of a slow leak or an undersized limit, even before it crosses any alert threshold. Comparing memory usage before and after each release catches this kind of regression at deploy time instead of at the next traffic spike.
|
||||
- **Synthetic load or canary checks after deploys.** Running a short load test against a new deployment before it takes full production traffic would have surfaced the higher memory footprint from the cache change immediately, rather than weeks later during an organic traffic spike.
|
||||
|
||||
Together, these close the two gaps that let this incident happen: no visibility into memory approaching its limit, and no mechanism that reacted to memory pressure before the orchestrator did.
|
||||
Together, these close the two gaps that let this incident happen: no visibility into memory approaching its limit, and no mechanism that reacted to memory pressure before the orchestrator did.
|
||||
|
||||
## 7. Early Detection with Ghaymah Tools
|
||||
|
||||
Ghaymah Cloud provides monitoring and logging capabilities that can help detect and investigate incidents such as the OOMKilled outage described in this postmortem. The platform provides application logs through the application dashboard, allowing engineers to inspect application behavior and troubleshoot failures. Ghaymah also provides resource usage monitoring through its Usage Dashboard, which can be used to observe resource consumption over time.
|
||||
|
||||
For this incident, engineers could use Ghaymah's monitoring and logs to identify abnormal resource consumption and investigate application failures following the deployment. Monitoring memory usage trends would help identify that the application's working-set memory was approaching or exceeding the available allocation.
|
||||
|
||||
However, the postmortem recommends adding proactive memory-threshold alerts and memory-aware autoscaling where supported by the underlying infrastructure. These controls would provide earlier warning and reduce the likelihood of repeated OOMKilled restarts.
|
||||
|
||||
The recommended detection flow is:
|
||||
|
||||
Monitor resource usage → Detect increasing memory consumption → Inspect application logs → Identify resource exhaustion → Increase allocated resources or scale the workload → Verify service recovery.
|
||||
المرجع في مشكلة جديدة
حظر مستخدم