Node clock drifts
NTP not synchronised, time offset grows.
ImpactTLS, tokens and distributed locks fail in strange ways
Can it be seen coming?Partly
RupturaNot covered yet
How to detect it
Offset above a threshold or sync lost.
Metrics and signals to watch:
node_timex_offset_secondsnode_timex_sync_status
Can you see it coming?
Partly Precursor: Offset growing after sync loss.
What it hits next
every pod of the node, etcd.
What to do
Fix time synchronisation on the node.
Sources
- node-exporter NodeClockNotSynchronising
- node-exporter NodeClockSkewDetected
Where Ruptura stands
Ruptura does not cover this failure yet. The card is here so you know it exists and how to detect it.
Related failures
- Node not ready
- Node flaps between Ready and NotReady
- Node under memory pressure
- Node under disk pressure
- Node under PID pressure
See it coming instead of reading about it
Ruptura is an open-source operator for Kubernetes and OpenShift that predicts the failures in this catalog that can be predicted, detects the others, and sends them to Alertmanager. Watch an OOM crash predicted 21 minutes early · ★ GitHub
Ruptura is an open-source operator for Kubernetes and OpenShift that predicts the failures in this catalog that can be predicted, detects the others, and sends them to Alertmanager. Watch an OOM crash predicted 21 minutes early · ★ GitHub