etcd disk too slow
WAL fsync and commit durations rise.
ImpactLeader elections, slow API, then an unavailable cluster
Can it be seen coming?Partly
RupturaNot covered yet
How to detect it
99th percentile above its threshold.
Metrics and signals to watch:
etcd_disk_wal_fsync_duration_secondsetcd_disk_backend_commit_duration_seconds
Can you see it coming?
Partly Precursor: Latency trending up with load.
What it hits next
the whole cluster.
What to do
Move etcd to faster disks or reduce load.
Sources
- etcd-mixin etcdHighFsyncDurations
- etcd-mixin etcdHighCommitDurations
Where Ruptura stands
Ruptura does not cover this failure yet. The card is here so you know it exists and how to detect it.
Related failures
- Admission webhook down or slow
- API server errors or slow requests
- etcd database reaches its quota
- etcd member down
See it coming instead of reading about it
Ruptura is an open-source operator for Kubernetes and OpenShift that predicts the failures in this catalog that can be predicted, detects the others, and sends them to Alertmanager. Watch an OOM crash predicted 21 minutes early · ★ GitHub
Ruptura is an open-source operator for Kubernetes and OpenShift that predicts the failures in this catalog that can be predicted, detects the others, and sends them to Alertmanager. Watch an OOM crash predicted 21 minutes early · ★ GitHub