We tried to predict Kubernetes failures before they happen. Here is every result, including the losses.
TL;DR
- Ruptura is an open source engine (Apache 2.0) that forecasts Kubernetes and OpenShift failures and gives an estimated time of failure.
- We compared it with what most teams already run: Prometheus predict_linear alert rules.
- Every test was written down before its data existed. Losses are published like wins.
- Headline: on a replica that leaks while its siblings stay healthy, Ruptura warned 8 minutes after the leak started and 57 minutes before the crash, against 30 minutes for predict_linear, with 0 false alarms.
- Memory failures over 12 hours: Ruptura warned on 65 of 73 leak cycles with a median lead of 15.4 minutes, against 38 of 73 and 7.0 minutes for the quiet predict_linear setting, with 0 false alarms on its trend forecasts, and on 14 of 14 periodic spikes no trend rule saw.
- Where it lost: a bad release that leaks fast (the rule warned earlier), two of six detection faults named under the wrong failure mode, and more findings than the rules on healthy workloads (real CPU throttling, but more of it).
Why this test
Every observability vendor says "predictive". Few show the numbers, and almost none show
them against the free baseline a team already has. So we set ourselves one rule: Ruptura must
beat a well-tuned predict_linear rule on the same cluster, the same faults, the same window,
or we say it does not.
The setup
- One OpenShift lab cluster, real workloads plus injected faults (memory leaks at several speeds, a periodic spike, crash loops, missing images and mounts, CPU throttling, a stuck rollout, a replica that diverges, a bad release).
- Ruptura 8.7.8, the published image, running with its default detection settings and frozen from 11 November (no restart during any scored window).
- Opponent: two
predict_linearsettings on container memory against its limit, a quiet one (30 min of history, 30 min ahead) and a fast one (10 min, 30 min), plus a rule set for the detection faults with their ownfordelays. - Rules written before the runs: what counts as a warning, what counts as false, how leads are measured, when a run is void. A void run is rerun once with the same rules, never rescored.
Scripts, rules and raw outputs: https://github.com/benfradjselim/ruptura/tree/main/benchmark (scripts published on 14 November, before the main benchmark day, byte-identical to the frozen versions).
Results
1. Memory failures, 12 hours (part A)
| Contestant | Trend leaks warned | Median fresh lead | Periodic spike | False episodes |
|---|---|---|---|---|
predict_linear 30m/30m |
38 of 73 | 7.0 min | 0 of 14 | 0 |
predict_linear 10m/30m |
73 of 73 | 21.5 min | 0 of 14 | 67 (61 on a healthy batch job) |
| Ruptura, trend forecasts only | 65 of 73 | 15.4 min | 0 of 14 | 0 |
| Ruptura with its failure memory | 73 of 73 | 18.3 min | 14 of 14 | 2 |
Verdict against the criteria written in advance: C1 pass (as many cycles and at least 5 min more lead than the quiet rule), C2 pass (at most half the false episodes of the fast rule, at most 2 more than the quiet one), C3 pass (at least 60 % of periodic spikes).
For an SRE: the quiet rule is calm but late, the fast rule is early but cries wolf 67 times in 12 hours on a healthy job. Ruptura sits where you want to be: earlier than the quiet rule, as calm, and it is the only one that sees a failure that comes back on a schedule. The fast rule still leads on plain trends; we say so.
2. A replica that diverges from its siblings (part F1, run 2026-11-13)
Three replicas, one starts leaking, three rounds.
| Ruptura | predict_linear (both settings) |
|
|---|---|---|
| Warned after leak start | 7.6 to 8.1 min | 34.5 to 35.5 min |
| Median lead before the OOM kill | 57.2 min | 29.8 min |
| Named the leaking pod | 3 of 3 | per-pod rule |
| False alarms | 0 | fired on 3 calm pods |
Why: Ruptura compares each pod with its own replicas, so a pod that grows while its siblings stay flat stands out long before it nears its limit. A trend on one series cannot see that. The same test run on Nov 4 gave the same picture (57.8 against 30.4 min).
3. A bad release (part F2)
A new version that leaks, rolled out beside a healthy version and a heavier but stable one, three rounds (run 2026-11-14). Ruptura lost this one on 2 of its 3 criteria.
| Ruptura | predict_linear (both settings) |
|
|---|---|---|
| Warned after the bad rollout | 11.2 to 12.2 min (criterion: within 15 min, met) | 1 to 2 min |
| Median lead before the OOM kill | 55.2 min | 64.8 min |
| False warnings | about 14 min of false warnings (heavy rollouts, rollbacks) | fired on 32 to 38 healthy pods |
What it means: a trend rule reacts to every new version's warm-up, so on a bad release it is first, and it is also wrong on almost every healthy rollout. Ruptura waits for evidence and catches the bad release within 12 minutes, but it was later than the rule here, and it still raised false warnings on the heavy version's warm-up and during rollbacks. Both are on our list.
4. Detection, 6 faults x 3 rounds (part B)
| Fault | Ruptura median | Rule median | Expected mode 3 of 3 |
|---|---|---|---|
| crash loop | 35 s | 1093 s | yes |
| image not found | 34 s | 968 s | yes |
| missing mount | 18 s | 968 s | yes |
| replicas unavailable | 1223 s | 968 s | no (listed as startup_stuck; loss stated before the run) |
| CPU throttling | 742 s | 1015 s | yes |
| OOM kill | 192 s | 667 s | no (flagged 8 min earlier, but as crash_loop or replicas_unavailable) |
| stuck rollout | 16 to 123 s after Kubernetes marks it | yes, 4 of 4 criteria |
5. Quiet, cost and setup (parts C, D, E)
- Findings on healthy workloads in 12 h: Ruptura 8 on 3 workloads, rules 5 on 2. All 8 are real CPU throttling (37 % to 99.9 % of CPU periods throttled), so 0 false, but the criterion was "no more findings than the rules": fail.
- Engine: 46 millicores CPU on average (96 at peak), 442 MiB memory at peak, close to its 512 MiB limit; we are looking at it.
- What a user writes to cover their namespaces: without Ruptura, 1 PrometheusRule with 8 rules (53 lines) plus 2 cluster settings (user workload monitoring, an Alertmanager route); with Ruptura, one RupturaInstance of 22 lines, 3 of them optional (console plugin).
Where Ruptura lost, and what we are doing about it
- Bad release (F2): the rule warned about 10 minutes earlier, and Ruptura raised false warnings on a heavy version's warm-up and during rollbacks. Fix in progress, to be measured under a new pre-registered test.
- Replicas unavailable: a Deployment with one unschedulable replica is named
startup_stuck(the Pending pod), notreplicas_unavailable, and 4 minutes later than the rule. We saw it in the dry run on 11 November and chose not to change the engine before the test. - OOM kill named wrong: Ruptura flagged the failing workload 8 minutes before the rule, but called it a crash loop. Detection is right, the label is not.
- More findings on healthy workloads: all real CPU throttling, but more than the rules raise. Whether throttling at that level deserves a finding is a product question we will settle openly.
Known limits we stated before the runs:
- The lab's faults are regular, which flatters the failure memory. Real clusters are noisier.
- The OOM rule has no for delay, so on OOM kill the best Ruptura can do is a tie.
- 99 of 125 pods on this lab have no memory limit. Forecasting against a limit cannot cover
them yet; node memory pressure is next on the roadmap.
Try it on your cluster
- OpenShift: OperatorHub, search "Ruptura" (https://operatorhub.io/operator/ruptura-operator).
- Kubernetes: Helm chart, see https://getruptura.dev.
- Demo video: https://youtu.be/sD1mMTWQgvY
- Code and issues: https://github.com/benfradjselim/ruptura
If you run it and it misses a failure, open an issue with what happened. That is the most useful thing you can send us. We are also looking for a few teams who want to run it on real clusters with us: become an early adopter.
We will present this live at Red Hat Summit Paris on 24 November.