New release leaks memory the previous one did not
After a rollout, the new version grows much faster than the previous one did.
ImpactOOM kills hours after a deploy that looked healthy
Can it be seen coming?Yes
RupturaCovered
How to detect it
Growth rate of the new version several times the previous one's.
Metrics and signals to watch:
container_memory_working_set_bytesReplicaSet of each pod
Can you see it coming?
Yes Precursor: The rate gap between versions, minutes after the rollout.
What it hits next
callers of the workload.
What to do
Roll back to the previous version or fix the new one.
Sources
- Kubernetes ReplicaSet revision
Where Ruptura stands
Ruptura reports this failure as the release_regression failure mode. A pre-registered lab test is not published yet.
Related failures
- Workload missing replicas
- StatefulSet missing replicas
- DaemonSet missing on some nodes
- Rollout passes its progress deadline
- Autoscaler at its maximum
See it coming instead of reading about it
Ruptura is an open-source operator for Kubernetes and OpenShift that predicts the failures in this catalog that can be predicted, detects the others, and sends them to Alertmanager. Watch an OOM crash predicted 21 minutes early · ★ GitHub
Ruptura is an open-source operator for Kubernetes and OpenShift that predicts the failures in this catalog that can be predicted, detects the others, and sends them to Alertmanager. Watch an OOM crash predicted 21 minutes early · ★ GitHub