Container crashes at start, again and again
The container exits shortly after start, Kubernetes backs off its restarts.
ImpactThe workload never serves
Can it be seen coming?No
RupturaCovered and tested on the lab
How to detect it
Event BackOff "restarting failed container", or repeated restarts.
Metrics and signals to watch:
Kubernetes event BackOffkube_pod_container_status_waiting_reasonkube_pod_container_status_restarts_total
Can you see it coming?
No Precursor: None before the first crash; a new release or configuration change is the usual cause. This failure arrives without a reliable warning sign, so the goal is to detect it fast and explain it.
What it hits next
callers of the workload.
What to do
Read the logs of the previous container and fix the startup failure.
Sources
- kubernetes-mixin KubePodCrashLooping
- Kubernetes waiting reason CrashLoopBackOff
Where Ruptura stands
Ruptura reports this failure as the crash_loop failure mode, and it is measured by a pre-registered test on the lab.
Related failures
- Memory leak ends in an OOM kill
- Memory growth accelerates toward the limit
- Sudden memory jump past the limit
- Container killed by OOM again and again
- Failures that come back at a regular interval
See it coming instead of reading about it
Ruptura is an open-source operator for Kubernetes and OpenShift that predicts the failures in this catalog that can be predicted, detects the others, and sends them to Alertmanager. Watch an OOM crash predicted 21 minutes early · ★ GitHub
Ruptura is an open-source operator for Kubernetes and OpenShift that predicts the failures in this catalog that can be predicted, detects the others, and sends them to Alertmanager. Watch an OOM crash predicted 21 minutes early · ★ GitHub