etcd database reaches its quota
The etcd database grows toward its space quota.
ImpactThe cluster becomes read-only
Can it be seen coming?Yes
RupturaPlanned
How to detect it
Size near the quota.
Metrics and signals to watch:
etcd_mvcc_db_total_size_in_bytesetcd_server_quota_backend_bytes
Can you see it coming?
Yes Precursor: Steady growth of the database, days ahead.
What it hits next
the whole cluster.
What to do
Compact and defragment, find what stores too many objects.
Sources
- etcd-mixin etcdDatabaseQuotaLowSpace
Where Ruptura stands
This failure is on the Ruptura roadmap. It is not reported yet.
Related failures
See it coming instead of reading about it
Ruptura is an open-source operator for Kubernetes and OpenShift that predicts the failures in this catalog that can be predicted, detects the others, and sends them to Alertmanager. Watch an OOM crash predicted 21 minutes early · ★ GitHub
Ruptura is an open-source operator for Kubernetes and OpenShift that predicts the failures in this catalog that can be predicted, detects the others, and sends them to Alertmanager. Watch an OOM crash predicted 21 minutes early · ★ GitHub