Node filesystem filling up
Free space on a node filesystem falls.
ImpactDiskPressure, evictions, then a node that cannot run pods
Can it be seen coming?Yes
RupturaPlanned
How to detect it
Free space below a threshold.
Metrics and signals to watch:
node_filesystem_avail_bytesnode_filesystem_size_bytes
Can you see it coming?
Yes Precursor: Steady fall of free space, hours or days ahead.
What it hits next
every pod of the node.
What to do
Find what writes, clean it, or grow the filesystem.
Sources
- node-exporter NodeFilesystemSpaceFillingUp
- node-exporter NodeFilesystemAlmostOutOfSpace
Where Ruptura stands
This failure is on the Ruptura roadmap. It is not reported yet.
Related failures
- Node not ready
- Node flaps between Ready and NotReady
- Node under memory pressure
- Node under disk pressure
- Node under PID pressure
See it coming instead of reading about it
Ruptura is an open-source operator for Kubernetes and OpenShift that predicts the failures in this catalog that can be predicted, detects the others, and sends them to Alertmanager. Watch an OOM crash predicted 21 minutes early · ★ GitHub
Ruptura is an open-source operator for Kubernetes and OpenShift that predicts the failures in this catalog that can be predicted, detects the others, and sends them to Alertmanager. Watch an OOM crash predicted 21 minutes early · ★ GitHub