DaemonSet missing on some nodes
Fewer DaemonSet pods available than nodes that should run it.
ImpactNodes without logging, networking or monitoring agents
Can it be seen coming?No
RupturaCovered
How to detect it
Desired minus available above zero for a sustained time.
Metrics and signals to watch:
kube_daemonset_status_desired_number_scheduledkube_daemonset_status_number_available
Can you see it coming?
No This failure arrives without a reliable warning sign, so the goal is to detect it fast and explain it.
What it hits next
pods of the affected nodes.
What to do
Check taints, resources and pod errors on the missing nodes.
Sources
- kubernetes-mixin KubeDaemonSetRolloutStuck
Where Ruptura stands
Ruptura reports this failure as the replicas_unavailable failure mode. A pre-registered lab test is not published yet.
Related failures
- Workload missing replicas
- StatefulSet missing replicas
- Rollout passes its progress deadline
- New release leaks memory the previous one did not
- Autoscaler at its maximum
See it coming instead of reading about it
Ruptura is an open-source operator for Kubernetes and OpenShift that predicts the failures in this catalog that can be predicted, detects the others, and sends them to Alertmanager. Watch an OOM crash predicted 21 minutes early · ★ GitHub
Ruptura is an open-source operator for Kubernetes and OpenShift that predicts the failures in this catalog that can be predicted, detects the others, and sends them to Alertmanager. Watch an OOM crash predicted 21 minutes early · ★ GitHub