The one replica that leaks while its siblings stay flat

Prefer video? The 4-minute overview above covers the whole idea.

Your Deployment runs five replicas. Four of them hold steady at 300 MiB. One climbs a little every minute.

On your dashboard, the average memory of the Deployment barely moves. Your alert on "memory above 90% of the limit" stays silent. Then that one pod is OOMKilled, restarts, and starts climbing again.

This is one of the failures that averages and global thresholds are built to miss. Here is how Ruptura, an open-source operator for Kubernetes and OpenShift, catches it, along with its close cousin: the release that leaks.

Why averages hide the leak

A Deployment-level metric is usually a sum or an average across pods. One replica growing while the others stay flat moves that number by a fifth of the growth. By the time it is visible, the pod is already close to its limit.

Looking at each pod against a fixed threshold does not help either: a pod at 70% of its limit can be perfectly healthy if that is where it always sits.

What matters is the rate, compared with the right reference.

Replica divergence: compare a pod with its siblings

Replicas of the same version run the same code on the same kind of traffic. They should grow at roughly the same rate. Ruptura uses that as the reference:

• It looks at versions with at least 3 replicas.
• When one replica grows at least 3 times faster than the others for 5 minutes, it raises a replica_divergence failure.
• The result names the pod, and is reported up to 2 hours ahead of the limit.
• When all replicas grow together, it stays silent: that is load, not a leak in one pod.

Release regression: compare a version with the one it replaced

The second pattern shows up right after a rollout. The new version leaks, every replica of it, while the previous version never did.

• After a 6 minute warm-up, Ruptura compares how fast the new version grows toward its memory limit with how fast the version it replaced did.
• When the new version grows at least 3 times faster, for 5 minutes, it raises a release_regression failure.
• It is reported up to 2 hours ahead, with both rates as evidence and a roll back as the recommended action.

That turns "memory looks high since this morning" into "the release you shipped at 9:40 leaks three times faster than the one before it; roll it back".

What you get

Each failure is one structured result, with the target, the failure mode, whether it is predicted or detected, the time left and the evidence. Ruptura sends it to your Alertmanager, so it follows the routes, silences and receivers you already use. On OpenShift it can use the cluster Alertmanager directly.

Both checks above are predictions: they fire before the pod is killed. Ruptura keeps those separate from detections such as oom_killed or crash_loop, which report what is already happening.

Try it

Locally, without a cluster:

git clone https://github.com/benfradjselim/ruptura
cd ruptura
docker compose up --build

On Kubernetes or OpenShift, the operator installs with two commands: see Getting started. The full list of failure modes and their rules is in failures.md.

Ruptura is open source (Apache 2.0) and still early. If you run Kubernetes or OpenShift with Prometheus, I am looking for teams to try it and tell me what is missing: become an early adopter. And if it is useful to you, a star on GitHub helps others find it.