ghaymah-exam-amirkasseb-sre/q2-postmortem/postmortem-report.md

Outage Postmortem: Repeated OOMKilled Events

1. Summary

On the day of the incident, a cloud-hosted application went down for 45 minutes. The root cause was repeated OOMKilled restarts: the container's memory usage exceeded its configured limit, the orchestrator killed the process, and the pod restarted in a loop instead of recovering. Users saw failed requests and timeouts for the full 45 minutes until the underlying memory limit was raised and the workload stabilized.

Severity: High

Duration: 45 minutes

Root cause: Container memory limit set below actual working-set memory, triggering a restart loop under load

Status: Resolved, follow-up actions listed below

2. Timeline

Time	Event
10:43 am	Traffic to the service increases beyond typical daily peak
10:53 am	First pod is OOMKilled; Kubernetes restarts it automatically
11:03 am	Memory usage climbs again post-restart, pod is killed a second time
11:08 am	Error rate alert fires; on-call engineer is paged
11:15 am	On-call confirms multiple pods are in CrashLoopBackOff
11:20 am	Logs and metrics dashboards confirm OOMKilled as the kill reason across all affected pods
11:23 am	Engineer raises the memory limit and requests for the deployment and applies the change
11:25 am	New pods start successfully and stay up under load
11:28 am	Error rate returns to baseline; incident closed

3. Root Cause Analysis

The service's memory limit had been set months earlier, based on load testing done before a feature update that added an in-memory cache. That cache grew with traffic and was never accounted for when the limit was set. Under normal daily traffic the service stayed under the limit. During a traffic spike, memory usage crossed the limit, the container was killed, and the fresh pod inherited the same undersized limit, so it hit the same wall again as soon as traffic resumed. This produced the crash loop.

Contributing factors:

• No memory-usage alert existed below the hard limit, so the team had no early warning before pods started dying.
• The horizontal pod autoscaler was configured on CPU usage only, not memory, so it did not add replicas to spread the load.
• Readiness and liveness probes were not tuned to fail fast on repeated restarts, which let the crash loop continue rather than triggering an automatic rollback.

4. Recommendations

1. Increase the memory limit and request to match current working-set usage plus headroom, based on the cache's real memory footprint under peak load.
2. Add a memory-usage alert at 75–80% of the limit so the team is warned before a pod is killed, not after.
3. Add memory as a second metric for the horizontal pod autoscaler, alongside CPU.
4. Set a restart-count threshold that triggers an automatic rollback to the previous deployment if a pod restarts more than a set number of times within a short window.
5. Load-test the service again after any change that adds in-memory state (caches, buffers, queues), since this is what caused the original limit to go stale.

5. Auto-Scaling Policy to Prevent Recurrence

Goal: scale on the resource that actually caused the incident (memory), not only on CPU, and react before pods are killed.

Policy design:

• Metrics used: CPU utilization and memory utilization, both tracked per pod against their requested values.
• Scale-out trigger: add replicas when either CPU exceeds 65% of request or memory exceeds 70% of the limit, sustained for 60 seconds. Using two metrics means a memory-bound spike triggers scaling even if CPU stays low, which is what happened during the incident.
• Scale-in trigger: remove replicas only when both CPU and memory stay below 40% of their configured values for 5 minutes, to avoid flapping.
• Minimum and maximum replicas: set a minimum that covers baseline traffic without cold starts, and a maximum sized to the platform's account or namespace resource quota, so scaling out cannot silently run into a quota wall during a spike.
• Pod Disruption Budget: configure a PDB so scale-in and node maintenance never drop available replicas below the minimum needed to serve traffic.
• Memory limit and request gap: keep a buffer between the memory request and the hard limit (for example, request at 70% of the limit) so a short-lived spike has room to breathe before the orchestrator kills the pod.
• Restart-based circuit breaker: if a pod restarts more than 3 times in 10 minutes, stop scaling that deployment further and page the on-call team instead, since more replicas of a broken configuration will not fix a bad memory limit.

This combination directly targets the incident: memory-aware scaling reacts to the same signal that caused the crash loop, the request/limit buffer gives pods room before they get killed, and the restart-based breaker stops the system from scaling into the same failure instead of alerting on it.

6. Early Detection with Cloud Monitoring Tools

The incident could have been caught well before the OOMKilled events with the following setup:

• Memory usage alerts below the hard limit. An alert at 75–80% of the container's memory limit gives the team a warning window instead of finding out only after a kill event. This is the single biggest gap in this incident: the limit existed, but nothing watched the approach to it.
• Restart and crash-loop alerts. Most cloud monitoring stacks can alert directly on container restart counts or on the OOMKilled reason code from the orchestrator. Alerting on the restart reason, not just "pod restarted," would have flagged this as a memory problem within the first minute.
• Dashboards that separate CPU and memory per workload. A single combined "resource usage" dashboard can hide a memory-only problem if CPU looks fine. Splitting the two, and showing them against their configured request/limit lines rather than raw numbers, makes a memory ceiling visible at a glance.
• Trend-based, not just threshold-based, alerts. A memory-usage graph that climbs steadily after each deploy is a leading indicator of a slow leak or an undersized limit, even before it crosses any alert threshold. Comparing memory usage before and after each release catches this kind of regression at deploy time instead of at the next traffic spike.
• Synthetic load or canary checks after deploys. Running a short load test against a new deployment before it takes full production traffic would have surfaced the higher memory footprint from the cache change immediately, rather than weeks later during an organic traffic spike.

Together, these close the two gaps that let this incident happen: no visibility into memory approaching its limit, and no mechanism that reacted to memory pressure before the orchestrator did.

7. Early Detection with Ghaymah Tools

Ghaymah Cloud provides monitoring and logging capabilities that can help detect and investigate incidents such as the OOMKilled outage described in this postmortem. The platform provides application logs through the application dashboard, allowing engineers to inspect application behavior and troubleshoot failures. Ghaymah also provides resource usage monitoring through its Usage Dashboard, which can be used to observe resource consumption over time.

For this incident, engineers could use Ghaymah's monitoring and logs to identify abnormal resource consumption and investigate application failures following the deployment. Monitoring memory usage trends would help identify that the application's working-set memory was approaching or exceeding the available allocation.

However, the postmortem recommends adding proactive memory-threshold alerts and memory-aware autoscaling where supported by the underlying infrastructure. These controls would provide earlier warning and reduce the likelihood of repeated OOMKilled restarts.

The recommended detection flow is:

Monitor resource usage → Detect increasing memory consumption → Inspect application logs → Identify resource exhaustion → Increase allocated resources or scale the workload → Verify service recovery.