Pod evicted for using too much local disk
Logs or scratch files fill the ephemeral storage, the kubelet evicts the pod.
ImpactThe pod restarts elsewhere and may fill that node too
Can it be seen coming?Yes
RupturaNot covered yet
How to detect it
Eviction message names ephemeral-storage.
Metrics and signals to watch:
Kubernetes event Evictedcontainer_fs_usage_bytes
Can you see it coming?
Yes Precursor: Local disk use growing toward its limit.
What it hits next
other pods of the node.
What to do
Rotate logs, clean scratch space, or set an ephemeral storage limit.
Sources
- Kubernetes event reason Evicted
Where Ruptura stands
Ruptura does not cover this failure yet. The card is here so you know it exists and how to detect it.
Related failures
- Image cannot be pulled
- Image name is not valid
- Volume cannot be mounted
- Volume cannot be attached to the node
- Pod cannot be scheduled
See it coming instead of reading about it
Ruptura is an open-source operator for Kubernetes and OpenShift that predicts the failures in this catalog that can be predicted, detects the others, and sends them to Alertmanager. Watch an OOM crash predicted 21 minutes early · ★ GitHub
Ruptura is an open-source operator for Kubernetes and OpenShift that predicts the failures in this catalog that can be predicted, detects the others, and sends them to Alertmanager. Watch an OOM crash predicted 21 minutes early · ★ GitHub