Network retransmits rise
TCP retransmits and receive errors grow on a node or path.
ImpactLatency and timeouts for everything crossing that path
Can it be seen coming?Partly
RupturaPlanned
How to detect it
Retransmit or error rate above its baseline.
Metrics and signals to watch:
node_netstat_Tcp_RetransSegsnode_network_receive_errs_total
Can you see it coming?
Partly Precursor: Errors growing on an interface.
What it hits next
pods of the node.
What to do
Check the interface, the MTU and the network fabric.
Sources
- node-exporter NodeNetworkReceiveErrs
- node-exporter NodeNetworkTransmitErrs
Where Ruptura stands
This failure is on the Ruptura roadmap. It is not reported yet.
Related failures
- Service with no ready endpoint
- Calls to a dependency fail
- DNS resolution fails
- NetworkPolicy blocks a needed call
- Route or Ingress has no healthy backend
See it coming instead of reading about it
Ruptura is an open-source operator for Kubernetes and OpenShift that predicts the failures in this catalog that can be predicted, detects the others, and sends them to Alertmanager. Watch an OOM crash predicted 21 minutes early · ★ GitHub
Ruptura is an open-source operator for Kubernetes and OpenShift that predicts the failures in this catalog that can be predicted, detects the others, and sends them to Alertmanager. Watch an OOM crash predicted 21 minutes early · ★ GitHub