Failure catalog · control-plane level

etcd disk too slow

WAL fsync and commit durations rise.

ImpactLeader elections, slow API, then an unavailable cluster
Can it be seen coming?Partly
RupturaNot covered yet

How to detect it

99th percentile above its threshold.

Metrics and signals to watch:

Can you see it coming?

Partly Precursor: Latency trending up with load.

What it hits next

the whole cluster.

What to do

Move etcd to faster disks or reduce load.

Sources

Where Ruptura stands

Ruptura does not cover this failure yet. The card is here so you know it exists and how to detect it.

Related failures

This card is part of an open catalog of 66 Kubernetes and OpenShift failures. Correct it or add one on GitHub.

See it coming instead of reading about it
Ruptura is an open-source operator for Kubernetes and OpenShift that predicts the failures in this catalog that can be predicted, detects the others, and sends them to Alertmanager. Watch an OOM crash predicted 21 minutes early · ★ GitHub