Predictive alert rules for Kubernetes and OpenShift

9 rules · free to copy · Apache 2.0 · validated PromQL

Most alert rules fire once something is already broken. These nine fire before: each one extrapolates a trend with predict_linear and warns while there is still time to act. Each links to its card in the open failure catalog.

AlertFires when the resource is on track toCard
ContainerMemoryWillHitLimithit its memory limit within 30 minutescard
PersistentVolumeWillFillUpfill up within 4 hourscard
NodeFilesystemWillFillUpfill up within 24 hourscard
NodeInodesWillRunOutrun out of inodes within 24 hourscard
NodeMemoryWillRunOutdrop below 5% available memory within 1 hourcard
NodeConntrackTableWillFillfill within 1 hourcard
ResourceQuotaWillBeExhaustedbe exhausted within 24 hourscard
HPAWillReachMaxReplicasreach its maximum replicas within 1 hourcard
EtcdDatabaseWillReachQuotareach its database quota within 3 dayscard

Install

Plain Prometheus: add the file to rule_files.

curl -sO https://getruptura.dev/rules/predictive-rules.yaml

Prometheus Operator, kube-prometheus or OpenShift: apply the PrometheusRule version in the namespace your Prometheus watches for rules.

curl -sO https://getruptura.dev/rules/predictive-prometheusrule.yaml
kubectl apply -n monitoring -f predictive-prometheusrule.yaml

On OpenShift, the rules that read node, kubelet, kube-state-metrics or etcd metrics need the platform Prometheus to evaluate them, not user workload monitoring. Check where your cluster evaluates platform rules before applying them.

Know the limits

A straight line is a rough forecast. These rules can miss a leak that accelerates, and fire on a burst that would have flattened out. Tune the windows to your workloads. Thresholds such as 15% or 20% free match common community practice, including the kubernetes-mixin volume rule.

Ruptura goes further: it names the failure, gives the time left with a window, shows the evidence and the services that go down with it, and compares each pod with its own history and its siblings. See an OOM crash predicted 21 minutes early.

The rules

# Predictive alert rules for Kubernetes and OpenShift
# From the open failure catalog: https://getruptura.dev/failures/
# Each rule extrapolates a trend with predict_linear and fires before the failure, not after.
# Free to copy and adapt (Apache 2.0). These are simple linear forecasts:
# Ruptura (https://getruptura.dev) adds failure names, time-left windows, evidence and blast radius.
groups:
  - name: predictive-failures
    rules:
      - alert: ContainerMemoryWillHitLimit
        expr: |
          max by (namespace, pod, container) (predict_linear(container_memory_working_set_bytes{container!="", container!="POD"}[30m], 30 * 60))
          > max by (namespace, pod, container) (kube_pod_container_resource_limits{resource="memory"})
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "{{ $labels.namespace }}/{{ $labels.pod }} ({{ $labels.container }}) is on track to hit its memory limit within 30 minutes"
          runbook_url: https://getruptura.dev/failures/memory-leak-to-oom/

      - alert: PersistentVolumeWillFillUp
        expr: |
          predict_linear(kubelet_volume_stats_available_bytes[6h], 4 * 3600) < 0
          and kubelet_volume_stats_available_bytes / kubelet_volume_stats_capacity_bytes < 0.15
        for: 15m
        labels:
          severity: warning
        annotations:
          summary: "PVC {{ $labels.namespace }}/{{ $labels.persistentvolumeclaim }} is on track to fill up within 4 hours"
          runbook_url: https://getruptura.dev/failures/persistent-volume-filling/

      - alert: NodeFilesystemWillFillUp
        expr: |
          predict_linear(node_filesystem_avail_bytes{fstype!~"tmpfs|overlay|squashfs"}[6h], 24 * 3600) < 0
          and node_filesystem_avail_bytes{fstype!~"tmpfs|overlay|squashfs"} / node_filesystem_size_bytes{fstype!~"tmpfs|overlay|squashfs"} < 0.2
        for: 30m
        labels:
          severity: warning
        annotations:
          summary: "{{ $labels.mountpoint }} on {{ $labels.instance }} is on track to fill up within 24 hours"
          runbook_url: https://getruptura.dev/failures/node-filesystem-filling/

      - alert: NodeInodesWillRunOut
        expr: |
          predict_linear(node_filesystem_files_free{fstype!~"tmpfs|overlay|squashfs"}[6h], 24 * 3600) < 0
          and node_filesystem_files_free{fstype!~"tmpfs|overlay|squashfs"} / node_filesystem_files{fstype!~"tmpfs|overlay|squashfs"} < 0.2
        for: 30m
        labels:
          severity: warning
        annotations:
          summary: "{{ $labels.mountpoint }} on {{ $labels.instance }} is on track to run out of inodes within 24 hours"
          runbook_url: https://getruptura.dev/failures/inode-exhaustion/

      - alert: NodeMemoryWillRunOut
        expr: |
          predict_linear(node_memory_MemAvailable_bytes[1h], 3600) < node_memory_MemTotal_bytes * 0.05
        for: 15m
        labels:
          severity: warning
        annotations:
          summary: "{{ $labels.instance }} is on track to drop below 5% available memory within 1 hour"
          runbook_url: https://getruptura.dev/failures/node-memory-pressure/

      - alert: NodeConntrackTableWillFill
        expr: |
          predict_linear(node_nf_conntrack_entries[1h], 3600) > node_nf_conntrack_entries_limit
        for: 15m
        labels:
          severity: warning
        annotations:
          summary: "The conntrack table on {{ $labels.instance }} is on track to fill within 1 hour"
          runbook_url: https://getruptura.dev/failures/conntrack-table-full/

      - alert: ResourceQuotaWillBeExhausted
        expr: |
          max by (namespace, resourcequota, resource) (predict_linear(kube_resourcequota{type="used"}[6h], 24 * 3600))
          > max by (namespace, resourcequota, resource) (kube_resourcequota{type="hard"} > 0)
        for: 30m
        labels:
          severity: info
        annotations:
          summary: "Quota {{ $labels.resourcequota }} ({{ $labels.resource }}) in {{ $labels.namespace }} is on track to be exhausted within 24 hours"
          runbook_url: https://getruptura.dev/failures/resource-quota-exhausted/

      - alert: HPAWillReachMaxReplicas
        expr: |
          predict_linear(kube_horizontalpodautoscaler_status_current_replicas[2h], 3600)
          >= kube_horizontalpodautoscaler_spec_max_replicas
          and kube_horizontalpodautoscaler_status_current_replicas < kube_horizontalpodautoscaler_spec_max_replicas
        for: 15m
        labels:
          severity: info
        annotations:
          summary: "HPA {{ $labels.namespace }}/{{ $labels.horizontalpodautoscaler }} is on track to reach its maximum replicas within 1 hour"
          runbook_url: https://getruptura.dev/failures/hpa-maxed-out/

      - alert: EtcdDatabaseWillReachQuota
        expr: |
          predict_linear(etcd_mvcc_db_total_size_in_bytes[24h], 3 * 24 * 3600) > etcd_server_quota_backend_bytes
        for: 1h
        labels:
          severity: warning
        annotations:
          summary: "etcd on {{ $labels.instance }} is on track to reach its database quota within 3 days"
          runbook_url: https://getruptura.dev/failures/etcd-database-quota/
Found a better rule?
Open an issue or a pull request on GitHub. · getruptura.dev