Rollout passes its progress deadline
New pods never become ready, Kubernetes sets Progressing to False.
ImpactThe new version is not served, the old one stays at reduced headroom
Can it be seen coming?No
RupturaCovered and tested on the lab
How to detect it
Progressing False with reason ProgressDeadlineExceeded.
Metrics and signals to watch:
kube_deployment_status_condition Progressing
Can you see it coming?
No This failure arrives without a reliable warning sign, so the goal is to detect it fast and explain it.
What it hits next
callers of the workload.
What to do
Inspect the new revision or roll back.
Sources
- kubernetes-mixin KubeDeploymentRolloutStuck
Where Ruptura stands
Ruptura reports this failure as the rollout_stuck failure mode, and it is measured by a pre-registered test on the lab.
Related failures
- Workload missing replicas
- StatefulSet missing replicas
- DaemonSet missing on some nodes
- New release leaks memory the previous one did not
- Autoscaler at its maximum
See it coming instead of reading about it
Ruptura is an open-source operator for Kubernetes and OpenShift that predicts the failures in this catalog that can be predicted, detects the others, and sends them to Alertmanager. Watch an OOM crash predicted 21 minutes early · ★ GitHub
Ruptura is an open-source operator for Kubernetes and OpenShift that predicts the failures in this catalog that can be predicted, detects the others, and sends them to Alertmanager. Watch an OOM crash predicted 21 minutes early · ★ GitHub