One replica grows while its siblings stay flat
Replicas of one version behave alike except one, whose memory climbs.
ImpactThat pod will be OOM killed, often from a bad node, a stuck connection or a skewed shard
Can it be seen coming?Yes
RupturaCovered and tested on the lab
How to detect it
One pod's growth rate several times its siblings'.
Metrics and signals to watch:
container_memory_working_set_byteskube_pod_container_resource_limitsReplicaSet of each pod
Can you see it coming?
Yes Precursor: The gap between the pod and its siblings.
What it hits next
requests routed to that pod.
What to do
Restart or delete the pod and check its node and state.
Sources
- Kubernetes ReplicaSet
Where Ruptura stands
Ruptura reports this failure as the replica_divergence failure mode, and it is measured by a pre-registered test on the lab.
Related failures
- Image cannot be pulled
- Image name is not valid
- Volume cannot be mounted
- Volume cannot be attached to the node
- Pod cannot be scheduled
See it coming instead of reading about it
Ruptura is an open-source operator for Kubernetes and OpenShift that predicts the failures in this catalog that can be predicted, detects the others, and sends them to Alertmanager. Watch an OOM crash predicted 21 minutes early · ★ GitHub
Ruptura is an open-source operator for Kubernetes and OpenShift that predicts the failures in this catalog that can be predicted, detects the others, and sends them to Alertmanager. Watch an OOM crash predicted 21 minutes early · ★ GitHub