Calls to a dependency fail
Connection refused, timeouts or resets toward one Service.
ImpactThe caller errors or queues
Can it be seen coming?Partly
RupturaPlanned
How to detect it
Error or reset rate toward one destination.
Metrics and signals to watch:
tracesservice mesh request metricskernel TCP events
Can you see it coming?
Partly Precursor: The failure of the dependency, when it is predicted.
What it hits next
callers of the caller.
What to do
Fix the dependency, add a timeout and a fallback.
Sources
- OpenTelemetry span status
- Istio request metrics
Where Ruptura stands
This failure is on the Ruptura roadmap. It is not reported yet.
Related failures
- Service with no ready endpoint
- DNS resolution fails
- Network retransmits rise
- NetworkPolicy blocks a needed call
- Route or Ingress has no healthy backend
See it coming instead of reading about it
Ruptura is an open-source operator for Kubernetes and OpenShift that predicts the failures in this catalog that can be predicted, detects the others, and sends them to Alertmanager. Watch an OOM crash predicted 21 minutes early · ★ GitHub
Ruptura is an open-source operator for Kubernetes and OpenShift that predicts the failures in this catalog that can be predicted, detects the others, and sends them to Alertmanager. Watch an OOM crash predicted 21 minutes early · ★ GitHub