Requests outgrow node allocatable
The sum of requests approaches what nodes can allocate.
ImpactNew pods unschedulable, autoscaling and rollouts stall
Can it be seen coming?Yes
RupturaPlanned
How to detect it
Requests over allocatable minus one node.
Metrics and signals to watch:
kube_pod_container_resource_requestskube_node_status_allocatable
Can you see it coming?
Yes Precursor: Requests growing week after week.
What it hits next
scheduling of new pods.
What to do
Add nodes or right-size requests.
Sources
- kubernetes-mixin KubeCPUOvercommit
- kubernetes-mixin KubeMemoryOvercommit
Where Ruptura stands
This failure is on the Ruptura roadmap. It is not reported yet.
Related failures
- Node not ready
- Node flaps between Ready and NotReady
- Node under memory pressure
- Node under disk pressure
- Node under PID pressure
See it coming instead of reading about it
Ruptura is an open-source operator for Kubernetes and OpenShift that predicts the failures in this catalog that can be predicted, detects the others, and sends them to Alertmanager. Watch an OOM crash predicted 21 minutes early · ★ GitHub
Ruptura is an open-source operator for Kubernetes and OpenShift that predicts the failures in this catalog that can be predicted, detects the others, and sends them to Alertmanager. Watch an OOM crash predicted 21 minutes early · ★ GitHub