Connection tracking table full
New connections dropped, kernel log reports a full conntrack table.
ImpactRandom connection failures for every pod of the node
Can it be seen coming?Yes
RupturaPlanned
How to detect it
Entries near the limit.
Metrics and signals to watch:
node_nf_conntrack_entriesnode_nf_conntrack_entries_limit
Can you see it coming?
Yes Precursor: Entries climbing with connection churn.
What it hits next
every pod of the node.
What to do
Raise the conntrack limit or reduce connection churn.
Sources
- node-exporter NodeHighNumberConntrackEntriesUsed
Where Ruptura stands
This failure is on the Ruptura roadmap. It is not reported yet.
Related failures
- Node not ready
- Node flaps between Ready and NotReady
- Node under memory pressure
- Node under disk pressure
- Node under PID pressure
See it coming instead of reading about it
Ruptura is an open-source operator for Kubernetes and OpenShift that predicts the failures in this catalog that can be predicted, detects the others, and sends them to Alertmanager. Watch an OOM crash predicted 21 minutes early · ★ GitHub
Ruptura is an open-source operator for Kubernetes and OpenShift that predicts the failures in this catalog that can be predicted, detects the others, and sends them to Alertmanager. Watch an OOM crash predicted 21 minutes early · ★ GitHub