etcd member down
A member stops answering, quorum shrinks.
ImpactOne more loss and the cluster stops
Can it be seen coming?No
RupturaNot covered yet
How to detect it
Member unreachable.
Metrics and signals to watch:
etcd_server_has_leaderup
Can you see it coming?
No This failure arrives without a reliable warning sign, so the goal is to detect it fast and explain it.
What it hits next
the whole cluster.
What to do
Restore the member before any other maintenance.
Sources
- etcd-mixin etcdMembersDown
- etcd-mixin etcdInsufficientMembers
Where Ruptura stands
Ruptura does not cover this failure yet. The card is here so you know it exists and how to detect it.
Related failures
- Admission webhook down or slow
- API server errors or slow requests
- etcd database reaches its quota
- etcd disk too slow
See it coming instead of reading about it
Ruptura is an open-source operator for Kubernetes and OpenShift that predicts the failures in this catalog that can be predicted, detects the others, and sends them to Alertmanager. Watch an OOM crash predicted 21 minutes early · ★ GitHub
Ruptura is an open-source operator for Kubernetes and OpenShift that predicts the failures in this catalog that can be predicted, detects the others, and sends them to Alertmanager. Watch an OOM crash predicted 21 minutes early · ★ GitHub