Failures that come back at a regular interval
The same workload fails every N minutes or hours.
ImpactPredictable outages, often from a batch, a leak or a scheduled job
Can it be seen coming?Yes
RupturaCovered and tested on the lab
How to detect it
Restart times with a stable interval.
Metrics and signals to watch:
kube_pod_container_status_restarts_total
Can you see it coming?
Yes Precursor: The interval of past failures.
What it hits next
callers of the workload.
What to do
Find what restarts the workload before the next failure.
Sources
- kubernetes-mixin KubePodCrashLooping
Where Ruptura stands
Ruptura reports this failure as the recurrent_failure failure mode, and it is measured by a pre-registered test on the lab.
Related failures
- Memory leak ends in an OOM kill
- Memory growth accelerates toward the limit
- Sudden memory jump past the limit
- Container killed by OOM again and again
- Container crashes at start, again and again
See it coming instead of reading about it
Ruptura is an open-source operator for Kubernetes and OpenShift that predicts the failures in this catalog that can be predicted, detects the others, and sends them to Alertmanager. Watch an OOM crash predicted 21 minutes early · ★ GitHub
Ruptura is an open-source operator for Kubernetes and OpenShift that predicts the failures in this catalog that can be predicted, detects the others, and sends them to Alertmanager. Watch an OOM crash predicted 21 minutes early · ★ GitHub