Job fails
The Job reaches its backoff limit.
ImpactBatch work not done, data not produced
Can it be seen coming?Partly
RupturaNot covered yet
How to detect it
Job condition Failed.
Metrics and signals to watch:
kube_job_failedkube_job_status_failed
Can you see it coming?
Partly Precursor: Duration or memory growing run after run.
What it hits next
consumers of the job's output.
What to do
Read the failed pod logs and fix the cause.
Sources
- kubernetes-mixin KubeJobFailed
Where Ruptura stands
Ruptura does not cover this failure yet. The card is here so you know it exists and how to detect it.
Related failures
- Workload missing replicas
- StatefulSet missing replicas
- DaemonSet missing on some nodes
- Rollout passes its progress deadline
- New release leaks memory the previous one did not
See it coming instead of reading about it
Ruptura is an open-source operator for Kubernetes and OpenShift that predicts the failures in this catalog that can be predicted, detects the others, and sends them to Alertmanager. Watch an OOM crash predicted 21 minutes early · ★ GitHub
Ruptura is an open-source operator for Kubernetes and OpenShift that predicts the failures in this catalog that can be predicted, detects the others, and sends them to Alertmanager. Watch an OOM crash predicted 21 minutes early · ★ GitHub