48 أسطر
2.0 KiB
Markdown
48 أسطر
2.0 KiB
Markdown
# Postmortem Report
|
|
|
|
## Summary
|
|
|
|
The downtime was caused by several occurrences of OOMKilled events, which lasted for 45 minutes. In this time frame, the application was down because the containers kept dying as they ran out of memory.
|
|
|
|
## Root Cause
|
|
|
|
The root cause of the problem is that the application uses more memory than allowed, causing the container to exceed the assigned limit of memory. This causes the Kubernetes to kill the container through OOMKilled actions resulting in 45 minutes of service unavailability.
|
|
|
|
## Timeline
|
|
## Note: This timeline is a simulation since exact time stamps have not been provided in the scenario.
|
|
|
|
| Time | Event |
|
|
| ----------- | -------------------------------- |
|
|
| 10:00 | Application started showing OOMKilled errors.
|
|
| 10:05 | Containers started to restart repeatedly, affecting services.
|
|
| 10:10 | Issue detected by monitoring systems; an alarm raised.
|
|
| 10:20 | Root cause of the problem detected; it is excessive memory consumption.
|
|
| 10:35 | Configuration for memory set; application restarted.
|
|
| 10:45 | Service restored to normal; service functioning fine.
|
|
|
|
## Recommendations
|
|
|
|
- Investigate and correct the problem of excessive memory use by the application.
|
|
- Set proper memory requests and limits for the container.
|
|
- Implement memory usage monitoring and alerting.
|
|
- Conduct load testing of the application to find out about memory problems ahead of time.
|
|
- Periodically analyze resource use by the application and optimize memory usage.
|
|
|
|
# Scaling Policy
|
|
|
|
- Minimum number of replicas: 2
|
|
- Maximum number of replicas: 10
|
|
- Scale out in case memory usage is above 70%.
|
|
- Scale in in case memory usage is below 40% for 5 minutes.
|
|
|
|
# Monitoring
|
|
|
|
This problem can be discovered earlier through monitoring:
|
|
- Memory usage in container.
|
|
- OOMKilled frequency.
|
|
- Pod restart count.
|
|
|
|
Notification to the team will be sent in case:
|
|
- Memory usage is above 80%.
|
|
- OOMKilled.
|
|
- Restart count is above a determined specific threshold. |