Container killed by OOM again and again
Repeated OOM kills, Kubernetes holds the container in CrashLoopBackOff.
ImpactThe workload is down most of the time
Can it be seen coming?Yes
RupturaCovered
How to detect it
CrashLoopBackOff with OOMKilled as last termination reason.
Metrics and signals to watch:
kube_pod_container_status_restarts_totalkube_pod_container_status_last_terminated_reason
Can you see it coming?
Yes Precursor: Regular interval between kills.
What it hits next
callers of the workload.
What to do
Raise the memory limit or find the memory spike before the next kill.
Sources
- kubernetes-mixin KubePodCrashLooping
Where Ruptura stands
Ruptura reports this failure as the recurrent_failure failure mode. A pre-registered lab test is not published yet.
Related failures
- Memory leak ends in an OOM kill
- Memory growth accelerates toward the limit
- Sudden memory jump past the limit
- Container crashes at start, again and again
- Failures that come back at a regular interval
See it coming instead of reading about it
Ruptura is an open-source operator for Kubernetes and OpenShift that predicts the failures in this catalog that can be predicted, detects the others, and sends them to Alertmanager. Watch an OOM crash predicted 21 minutes early · ★ GitHub
Ruptura is an open-source operator for Kubernetes and OpenShift that predicts the failures in this catalog that can be predicted, detects the others, and sends them to Alertmanager. Watch an OOM crash predicted 21 minutes early · ★ GitHub