Pod evicted under node memory pressure
The kubelet evicts pods when the node runs low on memory, first those above their requests.
ImpactPods without a limit die without any OOM at their own level
Can it be seen coming?Yes
RupturaPlanned
How to detect it
Event reason Evicted or node condition MemoryPressure.
Metrics and signals to watch:
Kubernetes event Evictedkube_node_status_condition MemoryPressurenode_memory_MemAvailable_bytes
Can you see it coming?
Yes Precursor: Node available memory falling while pods grow beyond their requests.
What it hits next
every pod of the node above its request.
What to do
Set requests and limits, or add memory to the node pool.
Sources
- Kubernetes node condition MemoryPressure
- Kubernetes event reason Evicted
Where Ruptura stands
This failure is on the Ruptura roadmap. It is not reported yet.
Related failures
- Image cannot be pulled
- Image name is not valid
- Volume cannot be mounted
- Volume cannot be attached to the node
- Pod cannot be scheduled
See it coming instead of reading about it
Ruptura is an open-source operator for Kubernetes and OpenShift that predicts the failures in this catalog that can be predicted, detects the others, and sends them to Alertmanager. Watch an OOM crash predicted 21 minutes early · ★ GitHub
Ruptura is an open-source operator for Kubernetes and OpenShift that predicts the failures in this catalog that can be predicted, detects the others, and sends them to Alertmanager. Watch an OOM crash predicted 21 minutes early · ★ GitHub