Failure catalog · pod level

One replica grows while its siblings stay flat

Replicas of one version behave alike except one, whose memory climbs.

ImpactThat pod will be OOM killed, often from a bad node, a stuck connection or a skewed shard
Can it be seen coming?Yes
RupturaCovered and tested on the lab

How to detect it

One pod's growth rate several times its siblings'.

Metrics and signals to watch:

Can you see it coming?

Yes Precursor: The gap between the pod and its siblings.

What it hits next

requests routed to that pod.

What to do

Restart or delete the pod and check its node and state.

Sources

Where Ruptura stands

Ruptura reports this failure as the replica_divergence failure mode, and it is measured by a pre-registered test on the lab.

Related failures

This card is part of an open catalog of 66 Kubernetes and OpenShift failures. Correct it or add one on GitHub.

See it coming instead of reading about it
Ruptura is an open-source operator for Kubernetes and OpenShift that predicts the failures in this catalog that can be predicted, detects the others, and sends them to Alertmanager. Watch an OOM crash predicted 21 minutes early · ★ GitHub