Liveness probe kills a slow container
The kubelet restarts a container whose liveness probe fails, often under load.
ImpactRestarts under load make the load worse
Can it be seen coming?Partly
RupturaNot covered yet
How to detect it
Unhealthy events for liveness followed by restarts.
Metrics and signals to watch:
Kubernetes event Unhealthykube_pod_container_status_restarts_total
Can you see it coming?
Partly Precursor: Probe latency rising with load or CPU throttling.
What it hits next
callers of the workload.
What to do
Relax the probe timeout or fix the slowness it measures.
Sources
- Kubernetes event reason Unhealthy
Where Ruptura stands
Ruptura does not cover this failure yet. The card is here so you know it exists and how to detect it.
Related failures
- Memory leak ends in an OOM kill
- Memory growth accelerates toward the limit
- Sudden memory jump past the limit
- Container killed by OOM again and again
- Container crashes at start, again and again
See it coming instead of reading about it
Ruptura is an open-source operator for Kubernetes and OpenShift that predicts the failures in this catalog that can be predicted, detects the others, and sends them to Alertmanager. Watch an OOM crash predicted 21 minutes early · ★ GitHub
Ruptura is an open-source operator for Kubernetes and OpenShift that predicts the failures in this catalog that can be predicted, detects the others, and sends them to Alertmanager. Watch an OOM crash predicted 21 minutes early · ★ GitHub